Package: lightly
Lightly SSL is a computer vision framework for self-supervised learning.DocumentationGithubDiscord(We have weekly paper sessions!)We've also built a whole platform on top, with additional features for active learning
anddata curation. If you're interested in the
Lightly Worker Solution to easily process millions of samples and runpowerful algorithmson your data, check outlightly.ai. It's free to get started!FeaturesThis self-supervised learning framework offers the following features:Modular framework, which exposes low-level building blocks such as loss functions and
model heads.Easy to use and written in a PyTorch like style.Supports custom backbone models for self-supervised pre-training.Support for distributed training using PyTorch Lightning.Supported ModelsYou canfind sample code for all the supported models here.We provide PyTorch, PyTorch Lightning,
and PyTorch Lightning distributed examples for all models to kickstart your project.Models:Barlow Twins, 2021paperdocsBYOL, 2020paperdocsDCL & DCLW, 2021paperdocsDINO, 2021paperdocsMAE, 2021paperdocsMSN, 2022paperdocsMoCo, 2019paperdocsNNCLR, 2021paperdocsPMSN, 2022paperdocsSimCLR, 2020paperdocsSimMIM, 2021paperdocsSimSiam, 2021paperdocsSMoG, 2022paperdocsSwaV, 2020paperdocsTiCo, 2022paperdocsVICReg, 2022paperdocsVICRegL, 2022paperdocsTutorialsWant to jump to the tutorials and see Lightly in action?Train MoCo on CIFAR-10Train SimCLR on Clothing DataTrain SimSiam on Satellite ImagesUse Lightly with Custom AugmentationsPre-train a Detectron2 Backbone with LightlyCommunity and partner projects:On-Device Deep Learning with Lightly on an ARM microcontrollerQuick StartLightly requiresPython 3.7+. We recommend installing Lightly in aLinuxorOSXenvironment. Python 3.12 is not yet supported, as PyTorch itself lacks Python 3.12 compatibility.DependenciesDue to the modular nature of the Lightly package some modules can be used with older versions of dependencies. However, to use all features as of today lightly requires the following dependencies:PyTorch>=1.11.0Torchvision>=0.12.0PyTorch Lightning>=1.7.1Lightly is compatible with PyTorch and PyTorch Lightning v2.0+!InstallationYou can install Lightly and its dependencies from PyPI with:pip3 install lightlyWe strongly recommend that you install Lightly in a dedicated virtualenv, to avoid
conflicting with your system packages.Lightly in ActionWith Lightly, you can use the latest self-supervised learning methods in a modular
way using the full power of PyTorch. Experiment with different backbones,
models, and loss functions. The framework has been designed to be easy to use
from the ground up.Find more examples in our docs.importtorchimporttorchvisionfromlightlyimportlossfromlightlyimporttransformsfromlightly.dataimportLightlyDatasetfromlightly.models.modulesimportheads# Create a PyTorch module for the SimCLR model.classSimCLR(torch.nn.Module):def__init__(self,backbone):super().__init__()self.backbone=backboneself.projection_head=heads.SimCLRProjectionHead(input_dim=512,# Resnet18 features have 512 dimensions.hidden_dim=512,output_dim=128,)defforward(self,x):features=self.backbone(x).flatten(start_dim=1)z=self.projection_head(features)returnz# Use a resnet backbone.backbone=torchvision.models.resnet18()# Ignore the classification head as we only want the features.backbone.fc=torch.nn.Identity()# Build the SimCLR model.model=SimCLR(backbone)# Prepare transform that creates multiple random views for every image.transform=transforms.SimCLRTransform(input_size=32,cj_prob=0.5)# Create a dataset from your image folder.dataset=LightlyDataset(input_dir="./my/cute/cats/dataset/",transform=transform)# Build a PyTorch dataloader.dataloader=torch.utils.data.DataLoader(dataset,# Pass the dataset to the dataloader.batch_size=128,# A large batch size helps with the learning.shuffle=True,# Shuffling is important!)# Lightly exposes building blocks such as loss functions.criterion=loss.NTXentLoss(temperature=0.5)# Get a PyTorch optimizer.optimizer=torch.optim.SGD(model.parameters(),lr=0.1,weight_decay=1e-6)# Train the model.forepochinrange(10):for(view0,view1),targets,filenamesindataloader:z0=model(view0)z1=model(view1)loss=criterion(z0,z1)loss.backward()optimizer.step()optimizer.zero_grad()print(f"loss:{loss.item():.5f}")You can easily use another model like SimSiam by swapping the model and the
loss function.# PyTorch module for the SimSiam model.classSimSiam(torch.nn.Module):def__init__(self,backbone):super().__init__()self.backbone=backboneself.projection_head=heads.SimSiamProjectionHead(512,512,128)self.prediction_head=heads.SimSiamPredictionHead(128,64,128)defforward(self,x):features=self.backbone(x).flatten(start_dim=1)z=self.projection_head(features)p=self.prediction_head(z)z=z.detach()returnz,pmodel=SimSiam(backbone)# Use the SimSiam loss function.criterion=loss.NegativeCosineSimilarity()You canfind a more complete example for SimSiam here.Use PyTorch Lightning to train the model:frompytorch_lightningimportLightningModule,TrainerclassSimCLR(LightningModule):def__init__(self):super().__init__()resnet=torchvision.models.resnet18()resnet.fc=torch.nn.Identity()self.backbone=resnetself.projection_head=heads.SimCLRProjectionHead(512,512,128)self.criterion=loss.NTXentLoss()defforward(self,x):features=self.backbone(x).flatten(start_dim=1)z=self.projection_head(features)returnzdeftraining_step(self,batch,batch_index):(view0,view1),_,_=batchz0=self.forward(view0)z1=self.forward(view1)loss=self.criterion(z0,z1)returnlossdefconfigure_optimizers(self):optim=torch.optim.SGD(self.parameters(),lr=0.06)returnoptimmodel=SimCLR()trainer=Trainer(max_epochs=10,devices=1,accelerator="gpu")trainer.fit(model,dataloader)Seeour docs for a full PyTorch Lightning example.Or train the model on 4 GPUs:# Use distributed version of loss functions.criterion=loss.NTXentLoss(gather_distributed=True)trainer=Trainer(max_epochs=10,devices=4,accelerator="gpu",strategy="ddp",sync_batchnorm=True,use_distributed_sampler=True,# or replace_sampler_ddp=True for PyTorch Lightning <2.0)trainer.fit(model,dataloader)We provide multi-GPU training examples with distributed gather and synchronized BatchNorm.Have a look at our docs regarding distributed training.BenchmarksImplemented models and their performance on various datasets. Hyperparameters are not
tuned for maximum accuracy. For detailed results and more information about the benchmarks clickhere.ImageNet1kImageNet1k benchmarksNote: Evaluation settings are based on these papers:Linear:SimCLRFinetune:SimCLRKNN:InstDiscSee thebenchmarking scriptsfor details.ModelBackboneBatch SizeEpochsLinear Top1Finetune Top1kNN Top1TensorboardCheckpointBarlowTwinsRes5025610062.972.645.6linklinkBYOLRes5025610062.474.045.6linklinkDINORes5012810068.272.549.9linklinkMoCoV2Res5025610061.574.441.3-linkSimCLR*Res5025610063.273.944.8linklinkSimCLR* + DCLRes5025610065.173.549.6linklinkSimCLR* + DCLWRes5025610064.573.248.5linklinkSwAVRes5025610067.275.449.5linklinkTiCoRes5025610049.772.726.6-linkVICRegRes5025610063.073.746.3linklink*We use square root learning rate scaling instead of linear scaling as it yields
better results for smaller batch sizes. See Appendix B.1 in theSimCLR paper.ImageNet100ImageNet100 benchmarks detailed resultsImagenetteImagenette benchmarks detailed resultsCIFAR-10CIFAR-10 benchmarks detailed resultsTerminologyBelow you can see a schematic overview of the different concepts in the package.
The terms in bold are explained in more detail in ourdocumentation.Next StepsHead to thedocumentationand see the things you can achieve with Lightly!DevelopmentTo install dev dependencies (for example to contribute to the framework) you can use the following command:pip3 install -e ".[dev]"For more information about how to contribute have a lookhere.Running TestsUnit tests are within thetests directoryand we recommend running them usingpytest. There are two test configurations
available. By default, only a subset will be run:make test-fastTo run all tests (including the slow ones) you can use the following command:make testTo test a specific file or directory use:pytest <path to file or directory>Code FormattingTo format code withblackandisortrun:make formatFurther ReadingSelf-Supervised Learning:Have a look at our#papers channel on discordfor the newest self-supervised learning papers.A Cookbook of Self-Supervised Learning, 2023Masked Autoencoders Are Scalable Vision Learners, 2021Emerging Properties in Self-Supervised Vision Transformers, 2021Unsupervised Learning of Visual Features by Contrasting Cluster Assignments, 2021What Should Not Be Contrastive in Contrastive Learning, 2020A Simple Framework for Contrastive Learning of Visual Representations, 2020Momentum Contrast for Unsupervised Visual Representation Learning, 2020FAQWhy should I care about self-supervised learning? Aren't pre-trained models from ImageNet much better for transfer learning?Self-supervised learning has become increasingly popular among scientists over the last years because the learned representations perform extraordinarily well on downstream tasks. This means that they capture the important information in an image better than other types of pre-trained models. By training a self-supervised model onyourdataset, you can make sure that the representations have all the necessary information about your images.How can I contribute?Create an issue if you encounter bugs or have ideas for features we should implement. You can also add your own code by forking this repository and creating a PR. More details about how to contribute with code is in ourcontribution guide.Is this framework for free?Yes, this framework is completely free to use and we provide the source code. We believe that we need to make training deep learning models more data efficient to achieve widespread adoption. One step to achieve this goal is by leveraging self-supervised learning. The company behind Lightly is committed to keep this framework open-source.If this framework is free, how is the company behind Lightly making money?Training self-supervised models is only one part of our solution.The company behind Lightlyfocuses on processing and analyzing embeddings created by self-supervised models.
By building, what we call a self-supervised active learning loop we help companies understand and work with their data more efficiently.
As theLightly Solutionis a freemium product, you can try it out for free. However, we will charge for some features.In any case this framework will always be free to use, even for commercial purposes.Lightly in ResearchReverse Engineering Self-Supervised Learning, 2023Learning Visual Representations via Language-Guided Sampling, 2023Self-Supervised Learning Methods for Label-Efficient Dental Caries Classification, 2022DPCL: Constrative Representation Learning with Differential Privacy, 2022Decoupled Contrastive Learning, 2021solo-learn: A Library of Self-supervised Methods for Visual Representation Learning, 2021Company behind this Open Source FrameworkLightlyis a spin-off from ETH Zurich that helps companies
build efficient active learning pipelines to select the most relevant data for their models.You can find out more about the company and it's services by following the links below:HomepageWeb-AppLightly Solution Documentation (Lightly Worker & API)BibTeXIf you want to cite the framework feel free to use this:@article{susmelj2020lightly,title={Lightly},author={Igor Susmelj and Matthias Heller and Philipp Wirth and Jeremy Prescott and Malte Ebner et al.},journal={GitHub. Note: https://github.com/lightly-ai/lightly},year={2020}}