Package: self-supervised
Self Supervised Learning with FastaiImplementation of popular SOTA self-supervised learning algorithms as Fastai Callbacks.Installpip install self-supervisedDocumentationPlease read the documentationhere.To go back to github repo please clickhere.AlgorithmsPlease read the papers or blog posts before getting started with an algorithm, you may also check out documentation page of each algorithm to get a better understanding.Here are the list of implementedself_supervised.visionalgorithms:SimCLR v1&SimCLR v2MoCo v1&MoCo v2BYOLSwAVBarlow TwinsDINOHere are the list of implementedself_supervised.multimodalalgorithms:CLIPCLIP-MoCo (No paper, own idea)For vision algorithms all models fromtimmandfastaican be used as encoders.For multimodal training currently CLIP supports ViT-B/32 and ViT-L/14, following best architectures from the paper.Simple UsageVisionSimCLRfromself_supervised.vision.simclrimport*dls=get_dls(resize,bs)# encoder = create_encoder("xresnet34", n_in=3, pretrained=False) # a fastai encoderencoder=create_encoder("tf_efficientnet_b4_ns",n_in=3,pretrained=False)# a timm encodermodel=create_simclr_model(encoder,hidden_size=2048,projection_size=128)aug_pipelines=get_simclr_aug_pipelines(size=size)learn=Learner(dls,model,cbs=[SimCLR(aug_pipelines,temp=0.07)])learn.fit_flat_cos(100,1e-2)MoCofromself_supervised.vision.mocoimport*dls=get_dls(resize,bs)# encoder = create_encoder("xresnet34", n_in=3, pretrained=False) # a fastai encoderencoder=create_encoder("tf_efficientnet_b4_ns",n_in=3,pretrained=False)# a timm encodermodel=create_moco_model(encoder,hidden_size=2048,projection_size=128)aug_pipelines=get_moco_aug_pipelines(size=size)learn=Learner(dls,model,cbs=[MOCO(aug_pipelines=aug_pipelines,K=128)])learn.fit_flat_cos(100,1e-2)BYOLfromself_supervised.vision.byolimport*dls=get_dls(resize,bs)# encoder = create_encoder("xresnet34", n_in=3, pretrained=False) # a fastai encoderencoder=create_encoder("tf_efficientnet_b4_ns",n_in=3,pretrained=False)# a timm encodermodel=create_byol_model(encoder,hidden_size=2048,projection_size=128)aug_pipelines=get_byol_aug_pipelines(size=size)learn=Learner(dls,model,cbs=[BYOL(aug_pipelines=aug_pipelines)])learn.fit_flat_cos(100,1e-2)SWAVfromself_supervised.vision.swavimport*dls=get_dls(resize,bs)encoder=create_encoder("xresnet34",n_in=3,pretrained=False)# a fastai encoderencoder=create_encoder("tf_efficientnet_b4_ns",n_in=3,pretrained=False)# a timm encodermodel=create_swav_model(encoder,hidden_size=2048,projection_size=128)aug_pipelines=get_swav_aug_pipelines(num_crops=[2,6],crop_sizes=[128,96],min_scales=[0.25,0.05],max_scales=[1.0,0.3])learn=Learner(dls,model,cbs=[SWAV(aug_pipelines=aug_pipelines,crop_assgn_ids=[0,1],K=bs*2**6,queue_start_pct=0.5)])learn.fit_flat_cos(100,1e-2)Barlow Twinsfromself_supervised.vision.simclrimport*dls=get_dls(resize,bs)# encoder = create_encoder("xresnet34", n_in=3, pretrained=False) # a fastai encoderencoder=create_encoder("tf_efficientnet_b4_ns",n_in=3,pretrained=False)# a timm encodermodel=create_barlow_twins_model(encoder,hidden_size=2048,projection_size=128)aug_pipelines=get_barlow_twins_aug_pipelines(size=size)learn=Learner(dls,model,cbs=[BarlowTwins(aug_pipelines,lmb=5e-3)])learn.fit_flat_cos(100,1e-2)DINOfromself_supervised.models.vision_transformerimport*fromself_supervised.vision.dinoimport*dls=get_dls(resize,bs)deits16=MultiCropWrapper(deit_small(patch_size=16,drop_path_rate=0.1))dino_head=DINOHead(deits16.encoder.embed_dim,2**16,norm_last_layer=True)student_model=nn.Sequential(deits16,dino_head)deits16=MultiCropWrapper(deit_small(patch_size=16))dino_head=DINOHead(deits16.encoder.embed_dim,2**16,norm_last_layer=True)teacher_model=nn.Sequential(deits16,dino_head)dino_model=DINOModel(student_model,teacher_model)aug_pipelines=get_dino_aug_pipelines(num_crops=[2,6],crop_sizes=[128,96],min_scales=[0.25,0.05],max_scales=[1.0,0.3])learn=Learner(dls,model,cbs=[DINO(aug_pipelines=aug_pipelines)])learn.fit_flat_cos(100,1e-2)MultimodalCLIPfromself_supervised.multimodal.clipimport*dls=get_dls(...)clip_tokenizer=ClipTokenizer()vitb32_config_dict=vitb32_config(224,clip_tokenizer.context_length,clip_tokenizer.vocab_size)clip_model=CLIP(**vitb32_config_dict,checkpoint=False,checkpoint_nchunks=0)learner=Learner(dls,clip_model,loss_func=noop,cbs=[CLIPTrainer()])learn.fit_flat_cos(100,1e-2)CLIP-MoCofromself_supervised.multimodal.clip_mocoimport*dls=get_dls(...)clip_tokenizer=ClipTokenizer()vitb32_config_dict=vitb32_config(224,clip_tokenizer.context_length,clip_tokenizer.vocab_size)clip_model=CLIPMOCO(K=4096,m=0.999,**vitb32_config_dict,checkpoint=False,checkpoint_nchunks=0)learner=Learner(dls,clip_model,loss_func=noop,cbs=[CLIPMOCOTrainer()])learn.fit_flat_cos(100,1e-2)ImageWang BenchmarksAll of the algorithms implemented in this library have been evaluated inImageWang Leaderboard.In overall superiority of the algorithms are as followsSwAV > MoCo > BYOL > SimCLRin most of the benchmarks. For details you may inspect the history ofImageWang Leaderboardthrough github.BarlowTwinsis still under testing on ImageWang.It should be noted that during these experiments no hyperparameter selection/tuning was made beyond usinglearn.lr_find()or making
sanity checks over data augmentations by visualizing batches. So, there is still space for improvement and overall rankings of the alogrithms may change based on your setup. Yet, the overall rankings are on par with the papers.ContributingContributions and or requests for new self-supervised algorithms are welcome. This repo will try to keep itself up-to-date with recent SOTA self-supervised algorithms.Before raising a PR please create a new branch with name<self-supervised-algorithm>. You may refer to previous notebooks before implementing your Callback.Please refer to sectionsDevelopers Guide, Abbreviations Guide, and Style Guidefromhttps://docs.fast.ai/dev-setupand note that same rules apply for this library.