Catalyst 101 — Accelerated PyTorch
By: Sergey Kolesnikov
Tags: Artificial Intelligence, Machine Learning, Deep Learning, Pytorch, Tutorial

Catalyst is rigorously tested and supports extra subpackages for Computer Vision and Natural Language Processing.

In addition, Catalyst.Team provides several production-ready pipelines,

catalyst-team/classification — comprehensive classification pipeline

catalyst-team/segmentation — binary and semantic segmentation pipeline

catalyst-team/detection — anchor-free detection pipeline

catalyst-team/video — video classification pipeline

catalyst-team/bert — DistilBERT pipeline for token classification tasks (archived version, updated release soon)

And extra packages for convenient Deep Learning R&D,

Getting started

But let’s make it simple, how could you start with Catalyst and join our Community? To begin with, let’s walk through MNIST classifier tutorial and compare PyTorch and Catalyst code side-by-side. While Catalyst supports arbitrarily complicated multi-stage deep learning pipelines, it’s still better to start with the basics.

Typical DL workflow

Each deep learning project typically consists of several main components:

the data

the model(s)

the optimizer(s)

and the loss(es)

The data

For this tutorial we’re using MNIST.

MNIST

Long story short, both Pytorch and Catalyst have the very same 4 steps for data preparation:

Download images Split them into train and validation datasets Add extra image transforms for splits (these are highly subjective) Wrap each dataset split into a DataLoader

Thanks to PyTorch team, 1–3 stages are already implemented for us:

At this step we have no differences between PyTorch and Catalyst code.

The model

Let’s design a 3-layer fully-connected neural network that takes as input an image that is 28x28 and outputs a logits distribution over 10 possible labels.

This model defines the computational graph to take as input an MNIST image and converts it to a logits distribution over 10 classes for digits 0–9.

Same here, we have no additional abstractions or custom modules, pure PyTorch code.

Btw, Catalyst also provides a wide variety of contrib models, for examples,

This way you can get the same model, but it’s a bit more extendable due to hiddens parameter.

The optimizer(s)

Now we choose how we’re going to do the optimization. For now, let’s use Adam as it is a good default in most DL projects.

Quite simple, isn’t it? It is as simple and powerful as PyTorch. Moreover, Catalyst provides a large variety of contrib Optimizers and easily handles multi-optimizer case, for example, for GANs experiments.

Catalyst works on top of PyTorch API and provides large number of extensions and user-friendly Experiment API to deliver “catalyzed” version of PyTorch. This ways you can boost your DL progress onto new level!

The loss(es)

Let’s use typical cross-entropy loss for our multi-class classification problem.

Again… the same code and full power of Catalyst contrib modules!

Training loop

And here we have all main components of our Experiment:

The dataset (Mnist) The model (3-layers NN) The optimizer The loss (criterion)

Now, let’s check how typical train loop looks like,

Deep Learning generalized training loop

Long story short, usually we

Go through our experiment stages (pretraining, training, finetuning)

Iterate for many epochs

Divide datasets in each iterated epoch into small samples of data — batches

Perform model forward pass

Compute the loss

Perform a backward pass to calculate all the gradients

Apply them to the model

And here we come to the Catalyst advantages. In PyTorch, you have to write the for loops yourself which means you have to remember to call the correct things in the right order — thinking more about engineering stuff, rather than your hypothesis testing.

Even if your model is simple, everything becomes complicated once you start doing more advanced things like multi-GPU or distributed training, gradient clipping, half-precision usage, early stopping, model checkpointing, etc…

And here comes the power of Catalyst. It abstracts the boilerplate and leaves main parts to you. You get a training loop with metrics, model checkpointing, advanced logging and distributed training support without the boilerplate. The same PyTorch code but much more readable and fully reproducible (another Catalyst feature).

Training Loop with PyTorch

The full MNIST example written in PyTorch is as follows:

Training Loop with Catalyst

The full MNIST example with Catalyst is exactly the same, but:

The training/validation loop code has been abstracted by the Runner

You have specified key point for both training and inference parts

You get automatic model validation and checkpointing

You get easy access to PyTorch tracing, distributed and half-precision training

Finally, you can focus only on key parts of your research

Catalyst extras

With Catalyst.dl and Catalyst.contrib the training loop can be simplified. You get access to

a variety of predefined metrics

a full-featured Callback mechanism

a bunch of Catalyst Loggers (Alchemy, Neptune, Tensorboard, Wandb)

a large number of contrib modules, supported by Catalyst Community

Deep Learning best practices, like `load_best_on_end` feature

Conclusion

Catalyst is pure PyTorch framework without any custom modules.

With Catalyst you get the same PyTorch pipelines but in much more readable and fully reproducible way.

Catalyst is easily extendable with Runner and Callback abstractions.

and abstractions. You focus on key parts of your deep learning R&D.

So, what are you waiting for?

Quick Start

In future tutorial series we will discuss more Catalyst-based examples, best practices and pipelines. Stay tuned!

More than a framework — Ecosystem