Package: confusionflow
ConfusionFlow is a visualization tool distributed as Python package that enables more nuanced monitoring of a neural network's training process.track and visualize the model performance of different timepoints (i.e., epochs) and dataset folds and compare different runs (e.g., different hyperparameter configurations).we provide wrappers for exporting performance logs in the required format.ConfusionFlow is in an early-preview alpha. Expect some bugs and rough edges.DemoTry the latest stable release of ConfusionFlow with severalexample logsfor the CIFAR-10 dataset:http://confusionflow.caleydoapp.orgAdditional InformationConfusionFlow was developed as a visualization tool to provide users with more feedback while developing or tuning neural network based classifiers. Performance monitoring often only utilizes simple line charts (one would for example plot the model loss and accuracy) which might easily miss many details and changes in the model's error structure.While the errors for a specific model state (e.g., at a certain epoch) can be represented as a confusion matrix, it is difficult to compare multiple confusion matrices or track changes over time.
ConfusionFlow visualizes model confusion over multiple model states and lets users compare different folds (e.g., train vs test set) or different hyperparameter configurations.To the best of our knowledge there are still no other tools with similar functionality.LimitationsNumber of classesDue to screenspace limitations the system is currently limited to around 10 classes. We are aware that we will not be able to handle datasets atImageNetscale BUT those datesets are not very common, as they are usually very expensive to obtain. A large percentage of classification problems range around 10 or less classes where ConfusionFlow can provide additional feedback.Runtime overheadLogging the performance for multiple folds every epoch might severely slowdown the time to convergence (especially when creating logs for the complete train and test sets). While logging on a minibatch level might provide additional information, it also severely slows down the training even further. It is possible to alleviate this problem by persisting model checkpoints and creating the performance logs on different machines. However, this also involves a lot of engineering effort and will not be supported by ConfusionFlow in the near future.InstallationConfusionflow can be either downloaded directly from PyPI viapip install confusionflowor by cloning and installing the repository directly from source:From SourceFirst make sure you havenodeinstalled (required for building the UI component).Clone and install the repository:git clone https://github.com/confusionflow/confusionflow
cd confusionflow
python setup.py installDevelopmentIf you want develop ConfusionFlow locally run:python setup.py build_ui developGetting StartedAs a first step you need to create some logs before you can start analyzing. Have a look at theexamplesfolder and run one of the examples.
The each example will create a new subdirectorylogswhere the performance logs will be stored.You then can start the ConfusionFlow UI via:confusionflow --logdir `<path_to_logdir>`Usage Examplestf.kerasfashion-mnisttorchmnistIf you are using your own datasets you have to create adataset-configurationfirst. We provide some example configurations for some popular datasets inexamples/dataset-templateswhich should help you getting started.DockerYou can also run ConfusionFlow via Docker. For more information please see thedocumentation.LoggingWe provide simple wrappers forTensorflow Keras APIas well asPytorchfor logging confusion matrices and exporting them in the format required by ConfusionFlow. For more information please consult thelogging documentation.Log Directory Layout<logdir>
├── datasets                    <--- dataset config files
│   ├── mnist.json
│   └── index.json
├── foldlogdata                 <--- foldlog data
│   ├── example_log_mnist_train_data.json
│   └── example_log_mnist_test_data.json
├── foldlog                     <--- foldlog specifications
│   ├── example_log_mnist_train.json
│   └── example_log_mnist_test.json
├── runs                        <--- run specifications
│   ├── example_log.json
│   └── index.json
└── views                       <--- view specifications (currently unsused)DocumentationThe latest documentation can be viewed ondocs.confusionflow.orgAPIThe Python package includes a simple Flask based server that implements the current API.
The current API definition can be found intools/swagger/api.ymland can be viewed using theSwagger Editor.ConfusionFlow UIThe ConfusionFlow UI is currently developed atCaleydo/confusionflow-ui.The TeamConfusionFlow is a research project of theInstitute of Computer GraphicsatJohannes Kepler University Linzin collaboration with theIBM Visual AI Lab.
ConfusionFlow is currently maintained byPeter RuchandHolger Stitz.FeedbackWe would be really grateful for anyfeedbackvia the repository's issues section.Getting InvolvedYou can ask questions on our mailing listconfusionflow-dev@googlegroups.com.Please report bugs by submitting aGitHub issue.