Package: catalyst
Accelerated Deep Learning R&DCatalyst is a PyTorch framework for Deep Learning Research and Development.
It focuses on reproducibility, rapid experimentation, and codebase reuse
so you can create something new rather than write yet another train loop.Break the cycle – use the Catalyst!Project ManifestFramework architectureCatalyst at AI LandscapePart of thePyTorch EcosystemCatalyst at PyTorch Ecosystem Day 2021Catalyst at PyTorch Developer Day 2021Getting startedpipinstall-Ucatalystimportosfromtorchimportnn,optimfromtorch.utils.dataimportDataLoaderfromcatalystimportdl,utilsfromcatalyst.contrib.datasetsimportMNISTmodel=nn.Sequential(nn.Flatten(),nn.Linear(28*28,10))criterion=nn.CrossEntropyLoss()optimizer=optim.Adam(model.parameters(),lr=0.02)loaders={"train":DataLoader(MNIST(os.getcwd(),train=True),batch_size=32),"valid":DataLoader(MNIST(os.getcwd(),train=False),batch_size=32),}runner=dl.SupervisedRunner(input_key="features",output_key="logits",target_key="targets",loss_key="loss")# model trainingrunner.train(model=model,criterion=criterion,optimizer=optimizer,loaders=loaders,num_epochs=1,callbacks=[dl.AccuracyCallback(input_key="logits",target_key="targets",topk=(1,3,5)),dl.PrecisionRecallF1SupportCallback(input_key="logits",target_key="targets"),],logdir="./logs",valid_loader="valid",valid_metric="loss",minimize_valid_metric=True,verbose=True,)# model evaluationmetrics=runner.evaluate_loader(loader=loaders["valid"],callbacks=[dl.AccuracyCallback(input_key="logits",target_key="targets",topk=(1,3,5))],)# model inferenceforpredictioninrunner.predict_loader(loader=loaders["valid"]):assertprediction["logits"].detach().cpu().numpy().shape[-1]==10# model post-processingmodel=runner.model.cpu()batch=next(iter(loaders["valid"]))[0]utils.trace_model(model=model,batch=batch)utils.quantize_model(model=model)utils.prune_model(model=model,pruning_fn="l1_unstructured",amount=0.8)utils.onnx_export(model=model,batch=batch,file="./logs/mnist.onnx",verbose=True)Step-by-step GuideStart withCatalyst — A PyTorch Framework for Accelerated Deep Learning R&Dintroduction.Trynotebook tutorialsor checkminimal examplesfor first deep dive.Readblog postswith use-cases and guides.Learn machine learning with our"Deep Learning with Catalyst" course.And finally,join our slackif you want to chat with the team and contributors.Table of ContentsGetting startedStep-by-step GuideTable of ContentsOverviewInstallationDocumentationMinimal ExamplesTestsBlog PostsTalksCommunityContribution GuideUser FeedbackAcknowledgmentsTrusted byCitationOverviewCatalyst helps you implement compact
but full-featured Deep Learning pipelines with just a few lines of code.
You get a training loop with metrics, early-stopping, model checkpointing,
and other features without the boilerplate.InstallationGeneric installation:pipinstall-UcatalystSpecialized versions, extra requirements might applypipinstallcatalyst[ml]# installs ML-based Catalystpipinstallcatalyst[cv]# installs CV-based Catalyst# master version installationpipinstallgit+https://github.com/catalyst-team/catalyst@master--upgrade# all available extensions are listed here:# https://github.com/catalyst-team/catalyst/blob/master/setup.pyCatalyst is compatible with: Python 3.7+. PyTorch 1.4+.Tested on Ubuntu 16.04/18.04/20.04, macOS 10.15, Windows 10, and Windows Subsystem for Linux.Documentationmaster22.022021 edition21.1221.1121.1021.0921.0821.0721.0621.05(Catalyst — A PyTorch Framework for Accelerated Deep Learning R&D)21.04/21.04.1,21.04.221.03,21.03.1/21.03.22020 edition20.1220.1120.1020.0920.08.220.07(dev blog: 20.07 release)20.0620.05,20.05.120.04,20.04.1,20.04.2Minimal ExamplesIntroduction tutorial "Customizing what happens intrain"Demo withcustomization examplesReinforcement Learning with CatalystAnd moreCustomRunner – PyTorch for-loop decompositionimportosfromtorchimportnn,optimfromtorch.nnimportfunctionalasFfromtorch.utils.dataimportDataLoaderfromcatalystimportdl,metricsfromcatalyst.contrib.datasetsimportMNISTmodel=nn.Sequential(nn.Flatten(),nn.Linear(28*28,10))optimizer=optim.Adam(model.parameters(),lr=0.02)train_data=MNIST(os.getcwd(),train=True)valid_data=MNIST(os.getcwd(),train=False)loaders={"train":DataLoader(train_data,batch_size=32),"valid":DataLoader(valid_data,batch_size=32),}classCustomRunner(dl.Runner):defpredict_batch(self,batch):# model inference stepreturnself.model(batch[0].to(self.engine.device))defon_loader_start(self,runner):super().on_loader_start(runner)self.meters={key:metrics.AdditiveMetric(compute_on_call=False)forkeyin["loss","accuracy01","accuracy03"]}defhandle_batch(self,batch):# model train/valid step# unpack the batchx,y=batch# run model forward passlogits=self.model(x)# compute the lossloss=F.cross_entropy(logits,y)# compute the metricsaccuracy01,accuracy03=metrics.accuracy(logits,y,topk=(1,3))# log metricsself.batch_metrics.update({"loss":loss,"accuracy01":accuracy01,"accuracy03":accuracy03})forkeyin["loss","accuracy01","accuracy03"]:self.meters[key].update(self.batch_metrics[key].item(),self.batch_size)# run model backward passifself.is_train_loader:self.engine.backward(loss)self.optimizer.step()self.optimizer.zero_grad()defon_loader_end(self,runner):forkeyin["loss","accuracy01","accuracy03"]:self.loader_metrics[key]=self.meters[key].compute()[0]super().on_loader_end(runner)runner=CustomRunner()# model trainingrunner.train(model=model,optimizer=optimizer,loaders=loaders,logdir="./logs",num_epochs=5,verbose=True,valid_loader="valid",valid_metric="loss",minimize_valid_metric=True,)# model inferenceforlogitsinrunner.predict_loader(loader=loaders["valid"]):assertlogits.detach().cpu().numpy().shape[-1]==10ML - linear regressionimporttorchfromtorch.utils.dataimportDataLoader,TensorDatasetfromcatalystimportdl# datanum_samples,num_features=int(1e4),int(1e1)X,y=torch.rand(num_samples,num_features),torch.rand(num_samples)dataset=TensorDataset(X,y)loader=DataLoader(dataset,batch_size=32,num_workers=1)loaders={"train":loader,"valid":loader}# model, criterion, optimizer, schedulermodel=torch.nn.Linear(num_features,1)criterion=torch.nn.MSELoss()optimizer=torch.optim.Adam(model.parameters())scheduler=torch.optim.lr_scheduler.MultiStepLR(optimizer,[3,6])# model trainingrunner=dl.SupervisedRunner()runner.train(model=model,criterion=criterion,optimizer=optimizer,scheduler=scheduler,loaders=loaders,logdir="./logdir",valid_loader="valid",valid_metric="loss",minimize_valid_metric=True,num_epochs=8,verbose=True,)ML - multiclass classificationimporttorchfromtorch.utils.dataimportDataLoader,TensorDatasetfromcatalystimportdl# sample datanum_samples,num_features,num_classes=int(1e4),int(1e1),4X=torch.rand(num_samples,num_features)y=(torch.rand(num_samples,)*num_classes).to(torch.int64)# pytorch loadersdataset=TensorDataset(X,y)loader=DataLoader(dataset,batch_size=32,num_workers=1)loaders={"train":loader,"valid":loader}# model, criterion, optimizer, schedulermodel=torch.nn.Linear(num_features,num_classes)criterion=torch.nn.CrossEntropyLoss()optimizer=torch.optim.Adam(model.parameters())scheduler=torch.optim.lr_scheduler.MultiStepLR(optimizer,[2])# model trainingrunner=dl.SupervisedRunner(input_key="features",output_key="logits",target_key="targets",loss_key="loss")runner.train(model=model,criterion=criterion,optimizer=optimizer,scheduler=scheduler,loaders=loaders,logdir="./logdir",num_epochs=3,valid_loader="valid",valid_metric="accuracy03",minimize_valid_metric=False,verbose=True,callbacks=[dl.AccuracyCallback(input_key="logits",target_key="targets",num_classes=num_classes),# uncomment for extra metrics:# dl.PrecisionRecallF1SupportCallback(#     input_key="logits", target_key="targets", num_classes=num_classes# ),# dl.AUCCallback(input_key="logits", target_key="targets"),# catalyst[ml] required ``pip install catalyst[ml]``# dl.ConfusionMatrixCallback(#     input_key="logits", target_key="targets", num_classes=num_classes# ),],)ML - multilabel classificationimporttorchfromtorch.utils.dataimportDataLoader,TensorDatasetfromcatalystimportdl# sample datanum_samples,num_features,num_classes=int(1e4),int(1e1),4X=torch.rand(num_samples,num_features)y=(torch.rand(num_samples,num_classes)>0.5).to(torch.float32)# pytorch loadersdataset=TensorDataset(X,y)loader=DataLoader(dataset,batch_size=32,num_workers=1)loaders={"train":loader,"valid":loader}# model, criterion, optimizer, schedulermodel=torch.nn.Linear(num_features,num_classes)criterion=torch.nn.BCEWithLogitsLoss()optimizer=torch.optim.Adam(model.parameters())scheduler=torch.optim.lr_scheduler.MultiStepLR(optimizer,[2])# model trainingrunner=dl.SupervisedRunner(input_key="features",output_key="logits",target_key="targets",loss_key="loss")runner.train(model=model,criterion=criterion,optimizer=optimizer,scheduler=scheduler,loaders=loaders,logdir="./logdir",num_epochs=3,valid_loader="valid",valid_metric="accuracy01",minimize_valid_metric=False,verbose=True,callbacks=[dl.BatchTransformCallback(transform=torch.sigmoid,scope="on_batch_end",input_key="logits",output_key="scores"),dl.AUCCallback(input_key="scores",target_key="targets"),# uncomment for extra metrics:# dl.MultilabelAccuracyCallback(input_key="scores", target_key="targets", threshold=0.5),# dl.MultilabelPrecisionRecallF1SupportCallback(#     input_key="scores", target_key="targets", threshold=0.5# ),])ML - multihead classificationimporttorchfromtorchimportnn,optimfromtorch.utils.dataimportDataLoader,TensorDatasetfromcatalystimportdl# sample datanum_samples,num_features,num_classes1,num_classes2=int(1e4),int(1e1),4,10X=torch.rand(num_samples,num_features)y1=(torch.rand(num_samples,)*num_classes1).to(torch.int64)y2=(torch.rand(num_samples,)*num_classes2).to(torch.int64)# pytorch loadersdataset=TensorDataset(X,y1,y2)loader=DataLoader(dataset,batch_size=32,num_workers=1)loaders={"train":loader,"valid":loader}classCustomModule(nn.Module):def__init__(self,in_features:int,out_features1:int,out_features2:int):super().__init__()self.shared=nn.Linear(in_features,128)self.head1=nn.Linear(128,out_features1)self.head2=nn.Linear(128,out_features2)defforward(self,x):x=self.shared(x)y1=self.head1(x)y2=self.head2(x)returny1,y2# model, criterion, optimizer, schedulermodel=CustomModule(num_features,num_classes1,num_classes2)criterion=nn.CrossEntropyLoss()optimizer=optim.Adam(model.parameters())scheduler=optim.lr_scheduler.MultiStepLR(optimizer,[2])classCustomRunner(dl.Runner):defhandle_batch(self,batch):x,y1,y2=batchy1_hat,y2_hat=self.model(x)self.batch={"features":x,"logits1":y1_hat,"logits2":y2_hat,"targets1":y1,"targets2":y2,}# model trainingrunner=CustomRunner()runner.train(model=model,criterion=criterion,optimizer=optimizer,scheduler=scheduler,loaders=loaders,num_epochs=3,verbose=True,callbacks=[dl.CriterionCallback(metric_key="loss1",input_key="logits1",target_key="targets1"),dl.CriterionCallback(metric_key="loss2",input_key="logits2",target_key="targets2"),dl.MetricAggregationCallback(metric_key="loss",metrics=["loss1","loss2"],mode="mean"),dl.BackwardCallback(metric_key="loss"),dl.OptimizerCallback(metric_key="loss"),dl.SchedulerCallback(),dl.AccuracyCallback(input_key="logits1",target_key="targets1",num_classes=num_classes1,prefix="one_"),dl.AccuracyCallback(input_key="logits2",target_key="targets2",num_classes=num_classes2,prefix="two_"),# catalyst[ml] required ``pip install catalyst[ml]``# dl.ConfusionMatrixCallback(#     input_key="logits1", target_key="targets1", num_classes=num_classes1, prefix="one_cm"# ),# dl.ConfusionMatrixCallback(#     input_key="logits2", target_key="targets2", num_classes=num_classes2, prefix="two_cm"# ),dl.CheckpointCallback(logdir="./logs/one",loader_key="valid",metric_key="one_accuracy01",minimize=False,topk=1),dl.CheckpointCallback(logdir="./logs/two",loader_key="valid",metric_key="two_accuracy03",minimize=False,topk=3),],loggers={"console":dl.ConsoleLogger(),"tb":dl.TensorboardLogger("./logs/tb")},)ML – RecSysimporttorchfromtorch.utils.dataimportDataLoader,TensorDatasetfromcatalystimportdl# sample datanum_users,num_features,num_items=int(1e4),int(1e1),10X=torch.rand(num_users,num_features)y=(torch.rand(num_users,num_items)>0.5).to(torch.float32)# pytorch loadersdataset=TensorDataset(X,y)loader=DataLoader(dataset,batch_size=32,num_workers=1)loaders={"train":loader,"valid":loader}# model, criterion, optimizer, schedulermodel=torch.nn.Linear(num_features,num_items)criterion=torch.nn.BCEWithLogitsLoss()optimizer=torch.optim.Adam(model.parameters())scheduler=torch.optim.lr_scheduler.MultiStepLR(optimizer,[2])# model trainingrunner=dl.SupervisedRunner(input_key="features",output_key="logits",target_key="targets",loss_key="loss")runner.train(model=model,criterion=criterion,optimizer=optimizer,scheduler=scheduler,loaders=loaders,num_epochs=3,verbose=True,callbacks=[dl.BatchTransformCallback(transform=torch.sigmoid,scope="on_batch_end",input_key="logits",output_key="scores"),dl.CriterionCallback(input_key="logits",target_key="targets",metric_key="loss"),# uncomment for extra metrics:# dl.AUCCallback(input_key="scores", target_key="targets"),# dl.HitrateCallback(input_key="scores", target_key="targets", topk=(1, 3, 5)),# dl.MRRCallback(input_key="scores", target_key="targets", topk=(1, 3, 5)),# dl.MAPCallback(input_key="scores", target_key="targets", topk=(1, 3, 5)),# dl.NDCGCallback(input_key="scores", target_key="targets", topk=(1, 3, 5)),dl.BackwardCallback(metric_key="loss"),dl.OptimizerCallback(metric_key="loss"),dl.SchedulerCallback(),dl.CheckpointCallback(logdir="./logs",loader_key="valid",metric_key="loss",minimize=True),])CV - MNIST classificationimportosfromtorchimportnn,optimfromtorch.utils.dataimportDataLoaderfromcatalystimportdlfromcatalyst.contrib.datasetsimportMNISTmodel=nn.Sequential(nn.Flatten(),nn.Linear(28*28,10))criterion=nn.CrossEntropyLoss()optimizer=optim.Adam(model.parameters(),lr=0.02)train_data=MNIST(os.getcwd(),train=True)valid_data=MNIST(os.getcwd(),train=False)loaders={"train":DataLoader(train_data,batch_size=32),"valid":DataLoader(valid_data,batch_size=32),}runner=dl.SupervisedRunner()# model trainingrunner.train(model=model,criterion=criterion,optimizer=optimizer,loaders=loaders,num_epochs=1,logdir="./logs",valid_loader="valid",valid_metric="loss",minimize_valid_metric=True,verbose=True,# uncomment for extra metrics:#     callbacks=[#         dl.AccuracyCallback(input_key="logits", target_key="targets", num_classes=10),#         dl.PrecisionRecallF1SupportCallback(#             input_key="logits", target_key="targets", num_classes=10#         ),#         dl.AUCCallback(input_key="logits", target_key="targets"),#         # catalyst[ml] required ``pip install catalyst[ml]``#         dl.ConfusionMatrixCallback(#             input_key="logits", target_key="targets", num_classes=num_classes#         ),#     ])CV - MNIST segmentationimportosimporttorchfromtorchimportnnfromtorch.utils.dataimportDataLoaderfromcatalystimportdlfromcatalyst.contrib.datasetsimportMNISTfromcatalyst.contrib.lossesimportIoULossmodel=nn.Sequential(nn.Conv2d(1,1,3,1,1),nn.ReLU(),nn.Conv2d(1,1,3,1,1),nn.Sigmoid(),)criterion=IoULoss()optimizer=torch.optim.Adam(model.parameters(),lr=0.02)train_data=MNIST(os.getcwd(),train=True)valid_data=MNIST(os.getcwd(),train=False)loaders={"train":DataLoader(train_data,batch_size=32),"valid":DataLoader(valid_data,batch_size=32),}classCustomRunner(dl.SupervisedRunner):defhandle_batch(self,batch):x=batch[self._input_key]x_noise=(x+torch.rand_like(x)).clamp_(0,1)x_=self.model(x_noise)self.batch={self._input_key:x,self._output_key:x_,self._target_key:x}runner=CustomRunner(input_key="features",output_key="scores",target_key="targets",loss_key="loss")# model trainingrunner.train(model=model,criterion=criterion,optimizer=optimizer,loaders=loaders,num_epochs=1,callbacks=[dl.IOUCallback(input_key="scores",target_key="targets"),dl.DiceCallback(input_key="scores",target_key="targets"),dl.TrevskyCallback(input_key="scores",target_key="targets",alpha=0.2),],logdir="./logdir",valid_loader="valid",valid_metric="loss",minimize_valid_metric=True,verbose=True,)CV - MNIST metric learningimportosfromtorch.optimimportAdamfromtorch.utils.dataimportDataLoaderfromcatalystimportdlfromcatalyst.contrib.dataimportHardTripletsSamplerfromcatalyst.contrib.datasetsimportMnistMLDataset,MnistQGDatasetfromcatalyst.contrib.lossesimportTripletMarginLossWithSamplerfromcatalyst.contrib.modelsimportMnistSimpleNetfromcatalyst.data.samplerimportBatchBalanceClassSampler# 1. train and valid loaderstrain_dataset=MnistMLDataset(root=os.getcwd())sampler=BatchBalanceClassSampler(labels=train_dataset.get_labels(),num_classes=5,num_samples=10,num_batches=10)train_loader=DataLoader(dataset=train_dataset,batch_sampler=sampler)valid_dataset=MnistQGDataset(root=os.getcwd(),gallery_fraq=0.2)valid_loader=DataLoader(dataset=valid_dataset,batch_size=1024)# 2. model and optimizermodel=MnistSimpleNet(out_features=16)optimizer=Adam(model.parameters(),lr=0.001)# 3. criterion with triplets samplingsampler_inbatch=HardTripletsSampler(norm_required=False)criterion=TripletMarginLossWithSampler(margin=0.5,sampler_inbatch=sampler_inbatch)# 4. training with catalyst RunnerclassCustomRunner(dl.SupervisedRunner):defhandle_batch(self,batch)->None:ifself.is_train_loader:images,targets=batch["features"].float(),batch["targets"].long()features=self.model(images)self.batch={"embeddings":features,"targets":targets,}else:images,targets,is_query=\batch["features"].float(),batch["targets"].long(),batch["is_query"].bool()features=self.model(images)self.batch={"embeddings":features,"targets":targets,"is_query":is_query}callbacks=[dl.ControlFlowCallbackWrapper(dl.CriterionCallback(input_key="embeddings",target_key="targets",metric_key="loss"),loaders="train",),dl.ControlFlowCallbackWrapper(dl.CMCScoreCallback(embeddings_key="embeddings",labels_key="targets",is_query_key="is_query",topk=[1],),loaders="valid",),dl.PeriodicLoaderCallback(valid_loader_key="valid",valid_metric_key="cmc01",minimize=False,valid=2),]runner=CustomRunner(input_key="features",output_key="embeddings")runner.train(model=model,criterion=criterion,optimizer=optimizer,callbacks=callbacks,loaders={"train":train_loader,"valid":valid_loader},verbose=False,logdir="./logs",valid_loader="valid",valid_metric="cmc01",minimize_valid_metric=False,num_epochs=10,)CV - MNIST GANimportosimporttorchfromtorchimportnnfromtorch.utils.dataimportDataLoaderfromcatalystimportdlfromcatalyst.contrib.datasetsimportMNISTfromcatalyst.contrib.layersimportGlobalMaxPool2d,Lambdalatent_dim=128generator=nn.Sequential(# We want to generate 128 coefficients to reshape into a 7x7x128 mapnn.Linear(128,128*7*7),nn.LeakyReLU(0.2,inplace=True),Lambda(lambdax:x.view(x.size(0),128,7,7)),nn.ConvTranspose2d(128,128,(4,4),stride=(2,2),padding=1),nn.LeakyReLU(0.2,inplace=True),nn.ConvTranspose2d(128,128,(4,4),stride=(2,2),padding=1),nn.LeakyReLU(0.2,inplace=True),nn.Conv2d(128,1,(7,7),padding=3),nn.Sigmoid(),)discriminator=nn.Sequential(nn.Conv2d(1,64,(3,3),stride=(2,2),padding=1),nn.LeakyReLU(0.2,inplace=True),nn.Conv2d(64,128,(3,3),stride=(2,2),padding=1),nn.LeakyReLU(0.2,inplace=True),GlobalMaxPool2d(),nn.Flatten(),nn.Linear(128,1),)model=nn.ModuleDict({"generator":generator,"discriminator":discriminator})criterion={"generator":nn.BCEWithLogitsLoss(),"discriminator":nn.BCEWithLogitsLoss()}optimizer={"generator":torch.optim.Adam(generator.parameters(),lr=0.0003,betas=(0.5,0.999)),"discriminator":torch.optim.Adam(discriminator.parameters(),lr=0.0003,betas=(0.5,0.999)),}train_data=MNIST(os.getcwd(),train=False)loaders={"train":DataLoader(train_data,batch_size=32)}classCustomRunner(dl.Runner):defpredict_batch(self,batch):batch_size=1# Sample random points in the latent spacerandom_latent_vectors=torch.randn(batch_size,latent_dim).to(self.engine.device)# Decode them to fake imagesgenerated_images=self.model["generator"](random_latent_vectors).detach()returngenerated_imagesdefhandle_batch(self,batch):real_images,_=batchbatch_size=real_images.shape[0]# Sample random points in the latent spacerandom_latent_vectors=torch.randn(batch_size,latent_dim).to(self.engine.device)# Decode them to fake imagesgenerated_images=self.model["generator"](random_latent_vectors).detach()# Combine them with real imagescombined_images=torch.cat([generated_images,real_images])# Assemble labels discriminating real from fake imageslabels=\torch.cat([torch.ones((batch_size,1)),torch.zeros((batch_size,1))]).to(self.engine.device)# Add random noise to the labels - important trick!labels+=0.05*torch.rand(labels.shape).to(self.engine.device)# Discriminator forwardcombined_predictions=self.model["discriminator"](combined_images)# Sample random points in the latent spacerandom_latent_vectors=torch.randn(batch_size,latent_dim).to(self.engine.device)# Assemble labels that say "all real images"misleading_labels=torch.zeros((batch_size,1)).to(self.engine.device)# Generator forwardgenerated_images=self.model["generator"](random_latent_vectors)generated_predictions=self.model["discriminator"](generated_images)self.batch={"combined_predictions":combined_predictions,"labels":labels,"generated_predictions":generated_predictions,"misleading_labels":misleading_labels,}runner=CustomRunner()runner.train(model=model,criterion=criterion,optimizer=optimizer,loaders=loaders,callbacks=[dl.CriterionCallback(input_key="combined_predictions",target_key="labels",metric_key="loss_discriminator",criterion_key="discriminator",),dl.BackwardCallback(metric_key="loss_discriminator"),dl.OptimizerCallback(optimizer_key="discriminator",metric_key="loss_discriminator",),dl.CriterionCallback(input_key="generated_predictions",target_key="misleading_labels",metric_key="loss_generator",criterion_key="generator",),dl.BackwardCallback(metric_key="loss_generator"),dl.OptimizerCallback(optimizer_key="generator",metric_key="loss_generator",),],valid_loader="train",valid_metric="loss_generator",minimize_valid_metric=True,num_epochs=20,verbose=True,logdir="./logs_gan",)# visualization (matplotlib required):# import matplotlib.pyplot as plt# %matplotlib inline# plt.imshow(runner.predict_batch(None)[0, 0].cpu().numpy())CV - MNIST VAEimportosimporttorchfromtorchimportnn,optimfromtorch.nnimportfunctionalasFfromtorch.utils.dataimportDataLoaderfromcatalystimportdl,metricsfromcatalyst.contrib.datasetsimportMNISTLOG_SCALE_MAX=2LOG_SCALE_MIN=-10defnormal_sample(loc,log_scale):scale=torch.exp(0.5*log_scale)returnloc+scale*torch.randn_like(scale)classVAE(nn.Module):def__init__(self,in_features,hid_features):super().__init__()self.hid_features=hid_featuresself.encoder=nn.Linear(in_features,hid_features*2)self.decoder=nn.Sequential(nn.Linear(hid_features,in_features),nn.Sigmoid())defforward(self,x,deterministic=False):z=self.encoder(x)bs,z_dim=z.shapeloc,log_scale=z[:,:z_dim//2],z[:,z_dim//2:]log_scale=torch.clamp(log_scale,LOG_SCALE_MIN,LOG_SCALE_MAX)z_=locifdeterministicelsenormal_sample(loc,log_scale)z_=z_.view(bs,-1)x_=self.decoder(z_)returnx_,loc,log_scaleclassCustomRunner(dl.IRunner):def__init__(self,hid_features,logdir,engine):super().__init__()self.hid_features=hid_featuresself._logdir=logdirself._engine=enginedefget_engine(self):returnself._enginedefget_loggers(self):return{"console":dl.ConsoleLogger(),"csv":dl.CSVLogger(logdir=self._logdir),"tensorboard":dl.TensorboardLogger(logdir=self._logdir),}@propertydefnum_epochs(self)->int:return1defget_loaders(self):loaders={"train":DataLoader(MNIST(os.getcwd(),train=False),batch_size=32),"valid":DataLoader(MNIST(os.getcwd(),train=False),batch_size=32),}returnloadersdefget_model(self):model=self.modelifself.modelisnotNoneelseVAE(28*28,self.hid_features)returnmodeldefget_optimizer(self,model):returnoptim.Adam(model.parameters(),lr=0.02)defget_callbacks(self):return{"backward":dl.BackwardCallback(metric_key="loss"),"optimizer":dl.OptimizerCallback(metric_key="loss"),"checkpoint":dl.CheckpointCallback(self._logdir,loader_key="valid",metric_key="loss",minimize=True,topk=3,),}defon_loader_start(self,runner):super().on_loader_start(runner)self.meters={key:metrics.AdditiveMetric(compute_on_call=False)forkeyin["loss_ae","loss_kld","loss"]}defhandle_batch(self,batch):x,_=batchx=x.view(x.size(0),-1)x_,loc,log_scale=self.model(x,deterministic=notself.is_train_loader)loss_ae=F.mse_loss(x_,x)loss_kld=(-0.5*torch.sum(1+log_scale-loc.pow(2)-log_scale.exp(),dim=1)).mean()loss=loss_ae+loss_kld*0.01self.batch_metrics={"loss_ae":loss_ae,"loss_kld":loss_kld,"loss":loss}forkeyin["loss_ae","loss_kld","loss"]:self.meters[key].update(self.batch_metrics[key].item(),self.batch_size)defon_loader_end(self,runner):forkeyin["loss_ae","loss_kld","loss"]:self.loader_metrics[key]=self.meters[key].compute()[0]super().on_loader_end(runner)defpredict_batch(self,batch):random_latent_vectors=torch.randn(1,self.hid_features).to(self.engine.device)generated_images=self.model.decoder(random_latent_vectors).detach()returngenerated_imagesrunner=CustomRunner(128,"./logs",dl.CPUEngine())runner.run()# visualization (matplotlib required):# import matplotlib.pyplot as plt# %matplotlib inline# plt.imshow(runner.predict_batch(None)[0].cpu().numpy().reshape(28, 28))AutoML - hyperparameters optimization with Optunaimportosimportoptunaimporttorchfromtorchimportnnfromtorch.utils.dataimportDataLoaderfromcatalystimportdlfromcatalyst.contrib.datasetsimportMNISTdefobjective(trial):lr=trial.suggest_loguniform("lr",1e-3,1e-1)num_hidden=int(trial.suggest_loguniform("num_hidden",32,128))train_data=MNIST(os.getcwd(),train=True)valid_data=MNIST(os.getcwd(),train=False)loaders={"train":DataLoader(train_data,batch_size=32),"valid":DataLoader(valid_data,batch_size=32),}model=nn.Sequential(nn.Flatten(),nn.Linear(784,num_hidden),nn.ReLU(),nn.Linear(num_hidden,10))optimizer=torch.optim.Adam(model.parameters(),lr=lr)criterion=nn.CrossEntropyLoss()runner=dl.SupervisedRunner(input_key="features",output_key="logits",target_key="targets")runner.train(model=model,criterion=criterion,optimizer=optimizer,loaders=loaders,callbacks={"accuracy":dl.AccuracyCallback(input_key="logits",target_key="targets",num_classes=10),# catalyst[optuna] required ``pip install catalyst[optuna]``"optuna":dl.OptunaPruningCallback(loader_key="valid",metric_key="accuracy01",minimize=False,trial=trial),},num_epochs=3,)score=trial.best_scorereturnscorestudy=optuna.create_study(direction="maximize",pruner=optuna.pruners.MedianPruner(n_startup_trials=1,n_warmup_steps=0,interval_steps=1),)study.optimize(objective,n_trials=3,timeout=300)print(study.best_value,study.best_params)Config API - minimal examplerunner:_target_:catalyst.runners.SupervisedRunnermodel:_var_:model_target_:torch.nn.Sequentialargs:-_target_:torch.nn.Flatten-_target_:torch.nn.Linearin_features:784# 28 * 28out_features:10input_key:featuresoutput_key:&output_keylogitstarget_key:&target_keytargetsloss_key:&loss_keylossrun:# ≈ stage 1-_call_:train# runner.train(...)criterion:_target_:torch.nn.CrossEntropyLossoptimizer:_target_:torch.optim.Adamparams:# model.parameters()_var_:model.parameterslr:0.02loaders:train:_target_:torch.utils.data.DataLoaderdataset:_target_:catalyst.contrib.datasets.MNISTroot:datatrain:ybatch_size:32&valid_loader_key valid:&valid_loader_target_:torch.utils.data.DataLoaderdataset:_target_:catalyst.contrib.datasets.MNISTroot:datatrain:nbatch_size:32callbacks:-&accuracy_metric_target_:catalyst.callbacks.AccuracyCallbackinput_key:*output_keytarget_key:*target_keytopk:[1,3,5]-_target_:catalyst.callbacks.PrecisionRecallF1SupportCallbackinput_key:*output_keytarget_key:*target_keynum_epochs:1logdir:logsvalid_loader:*valid_loader_keyvalid_metric:*loss_keyminimize_valid_metric:yverbose:y# ≈ stage 2-_call_:evaluate_loader# runner.evaluate_loader(...)loader:*valid_loadercallbacks:-*accuracy_metriccatalyst-run--configexample.yamlTestsAll Catalyst code, features, and pipelinesare fully tested.
We also have our owncatalyst-codestyleand a corresponding pre-commit hook.
During testing, we train a variety of different models: image classification,
image segmentation, text classification, GANs, and much more.
We then compare their convergence metrics in order to verify
the correctness of the training procedure and its reproducibility.
As a result, Catalyst provides fully tested and reproducible
best practices for your deep learning research and development.Blog PostsTalksCommunityAccelerated with CatalystResearch PapersHierarchical Attention for Sentiment Classification with VisualizationPediatric Bone Age AssessmentImplementation of the paper "Tell Me Where to Look: Guided Attention Inference Network"Implementation of the paper "Filter Response Normalization Layer: Eliminating Batch Dependence in the Training of Deep Neural Networks"Implementation of the paper "Utterance-level Aggregation For Speaker Recognition In The Wild"Implementation of the paper "Looking to Listen at the Cocktail Party: A Speaker-Independent Audio-Visual Model for Speech Separation"Implementation of the paper "ESRGAN: Enhanced Super-Resolution Generative Adversarial Networks"Blog PostsSolving the Cocktail Party Problem using PyTorchBeyond fashion: Deep Learning with Catalyst (Config API)Tutorial from Notebook API to Config API (RU)CompetitionsKaggle Quick, Draw! Doodle Recognition Challenge- 11th placeCatalyst.RL - NeurIPS 2018: AI for Prosthetics Challenge– 3rd placeKaggle Google Landmark 2019- 30th placeiMet Collection 2019 - FGVC6- 24th placeID R&D Anti-spoofing Challenge- 14th placeNeurIPS 2019: Recursion Cellular Image Classification- 4th placeMICCAI 2019: Automatic Structure Segmentation for Radiotherapy Planning Challenge 20193rd place solution forTask 3: Organ-at-risk segmentation from chest CT scansand 4th place solution forTask 4: Gross Target Volume segmentation of lung cancerKaggle Seversteal steel detection- 5th placeRSNA Intracranial Hemorrhage Detection- 5th placeAPTOS 2019 Blindness Detection– 7th placeCatalyst.RL - NeurIPS 2019: Learn to Move - Walk Around– 2nd placexView2 Damage Assessment Challenge- 3rd placeToolkitsCatalyst.RL– A Distributed Framework for Reproducible RL Research byScitatorCatalyst.Classification- Comprehensive classification pipeline with Pseudo-Labeling byBagxiandPdanilovCatalyst.Segmentation- Segmentation pipelines - binary, semantic and instance, byBagxiCatalyst.Detection- Anchor-free detection pipeline byAvi2011classandTezRomacHCatalyst.GAN- Reproducible GANs pipelines byAsmekalCatalyst.Neuro- Brain image analysis project, in collaboration withTReNDS CenterMLComp– Distributed DAG framework for machine learning with UI byLightforeverPytorch toolbelt- PyTorch extensions for fast R&D prototyping and Kaggle farming byBloodAxeHelper functions- An assorted collection of helper functions byTernausBERT Distillation with CatalystbyelephantmiptOtherCamVid Segmentation Example- Example of semantic segmentation for CamVid datasetNotebook API tutorial for segmentation in Understanding Clouds from Satellite Images CompetitionCatalyst.RL - NeurIPS 2019: Learn to Move - Walk Around– starter kitCatalyst.RL - NeurIPS 2019: Animal-AI Olympics- starter kitInria Segmentation Example- An example of training segmentation model for Inria Sattelite Segmentation Challengeiglovikov_segmentation- Semantic segmentation pipeline using CatalystLogging Catalyst Runs to Comet- An example of how to log metrics, hyperparameters and more from Catalyst runs toCometSee other projects atthe GitHub dependency graph.If your project implements a paper,
a notable use-case/tutorial, or a Kaggle competition solution, or
if your code simply presents interesting results and uses Catalyst,
we would be happy to add your project to the list above!
Do not hesitate to send us a PR with a brief description of the project similar to the above.Contribution GuideWe appreciate all contributions.
If you are planning to contribute back bug-fixes, there is no need to run that by us; just send a PR.
If you plan to contribute new features, new utility functions, or extensions,
please open an issue first and discuss it with us.Please see theContribution Guidefor more information.By participating in this project, you agree to abide by itsCode of Conduct.User FeedbackWe've createdfeedback@catalyst-team.comas an additional channel for user feedback.If you like the project and want to thank us, this is the right place.If you would like to start a collaboration between your team and Catalyst team to improve Deep Learning R&D, you are always welcome.If you don't like Github Issues and prefer email, feel free to email us.Finally, if you do not like something, please, share it with us, and we can see how to improve it.We appreciate any type of feedback. Thank you!AcknowledgmentsSince the beginning of the Сatalyst development, a lot of people have influenced it in a lot of different ways.Catalyst.TeamDmytro Doroshenko(ditwoo)Eugene Kachan(bagxi)Nikita Balagansky(elephantmipt)Sergey Kolesnikov(scitator)Catalyst.ContributorsAleksey Grinchuk(alexgrinch)Aleksey Shabanov(AlekseySh)Alex Gaziev(gazay)Andrey Zharkov(asmekal)Artem Zolkin(arquestro)David Kuryakin(dkuryakin)Evgeny Semyonov(lightforever)Eugene Khvedchenya(bloodaxe)Ivan StepanenkoJulia Shenshina(julia-shenshina)Nguyen Xuan Bac(ngxbac)Roman Tezikov(TezRomacH)Valentin Khrulkov(khrulkovv)Vladimir Iglovikov(ternaus)Vsevolod Poletaev(hexfaker)Yury Kashnitsky(yorko)Trusted byAwecomResearchers at theCenter for Translational Research in Neuroimaging and Data Science (TReNDS)Deep Learning SchoolResearchers atEmory UniversityEvil MartiansResearchers at theGeorgia Institute of TechnologyResearchers atGeorgia State UniversityHeliosHPCD LabiFarmKinoplanResearchers at theMoscow Institute of Physics and TechnologyNeuromationPoteha LabsProvectusResearchers at theSkolkovo Institute of Science and TechnologySoftConstructResearchers atTinkoffResearchers atYandex.ResearchCitationPlease use this bibtex if you want to cite this repository in your publications:@misc{catalyst,
    author = {Kolesnikov, Sergey},
    title = {Catalyst - Accelerated deep learning R&D},
    year = {2018},
    publisher = {GitHub},
    journal = {GitHub repository},
    howpublished = {\url{https://github.com/catalyst-team/catalyst}},
}