Package: glasses
%load_extautoreload%autoreload2Glasses 😎Compact, concise and customizable
deep learning computer vision librarySo far I have thefollowingpretrainde weights. I am working on porting more. They are hosted on GitHub if < 100MB and on AWS (thaks to Francis Ukpeh) if > 100MB.*Doc ishereTL;TRThis library hashuman readable code, noresearch codecommon component are shared acrossmodelssame APIsfor all models (you learn them once and they are always the same)clear and easy to use model constomization (seehere)classificationandsegmentationemoji in the name ;)Architectures implemented so far:Training data-efficient image transformers & distillation through attentionVision Transformer -  An Image Is Worth 16x16 Words: Transformers For Image Recognition At ScaleResNeSt: Split-Attention NetworksAlexNet-  ImageNet Classification with Deep Convolutional Neural NetworksDenseNet - Densely Connected Convolutional NetworksEfficientNet - EfficientNet: Rethinking Model Scaling for Convolutional Neural NetworksEfficientNetLite - Higher accuracy on vision models with EfficientNet-LiteFishNet - FishNet: A Versatile Backbone for Image, Region, and Pixel Level PredictionMobileNet - MobileNetV2: Inverted Residuals and Linear BottlenecksRegNet - Designing Network Design SpacesResNet - Deep Residual Learning for Image RecognitionResNetD - Bag of Tricks for Image Classification with Convolutional Neural NetworksResNetXt - Aggregated Residual Transformations for Deep Neural NetworksSEResNet - Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional NetworksVGG - Very Deep Convolutional Networks For Large-scale Image RecognitionWideResNet - Wide Residual NetworksFPN - Feature Pyramid Networks for Object DetectionPFPN - Panoptic Feature Pyramid NetworksUNet - U-Net: Convolutional Networks for Biomedical Image SegmentationSqueeze and Excitation - Concurrent Spatial and Channel Squeeze & Excitation in Fully Convolutional NetworksECA - ECA-Net: Efficient Channel Attention for Deep Convolutional Neural NetworksInstallationYou can installglassesusing pip by runningpip install git+https://github.com/FrancescoSaverioZuppichini/glassesMotivationsAlmost all existing implementations of the most famous model are written with very bad coding practices, what today is calledresearch code. I struggled myself to understand some of the implementations that in the end were just a few lines of code.Most of them are missing a global structure, they used tons of code repetition, they are not easily customizable and not tested. Since I do computer vision for living, so I needed a way to make my life easier.Getting startedThe API are shared acrossallmodels!importtorchfromglasses.modelsimportAutoModel,AutoConfigfromtorchimportnn# load one modelmodel=AutoModel.from_pretrained('resnet18')cfg=AutoConfig.from_name('resnet18')model.summary(device='cpu')# thanks to torchsummaryAutoModel.models()# 'resnet18', 'resnet26', 'resnet26d', 'resnet34', 'resnet50', ...InterpretabilityimportrequestsfromPILimportImagefromioimportBytesIOfromglasses.interpretabilityimportGradCam,SaliencyMapfromtorchvision.transformsimportNormalizer=requests.get('https://i.insider.com/5df126b679d7570ad2044f3e?width=700&format=jpeg&auto=webp')im=Image.open(BytesIO(r.content))# un normalize when donepostprocessing=Normalize(-cfg.mean/cfg.std,(1.0/cfg.std))# apply preprocessingx=cfg.transform(im).unsqueeze(0)_=model.interpret(x,using=GradCam(),postprocessing=postprocessing).show()Classificationfromglasses.modelsimportResNet# change activationResNet.resnet18(activation=nn.SELU)# change number of classesResNet.resnet18(n_classes=100)# freeze only the convolution weightsmodel=ResNet.resnet18(pretrained=True)model.freeze(who=model.encoder)# get the last layer, usuful to hook to it if you want to get the embeeded vectormodel.encoder.layers[-1]# what about resnet with inverted residuals?fromglasses.models.classification.efficientnetimportInvertedResidualBlockResNet.resnet18(block=InvertedResidualBlock)Segmentationfromfunctoolsimportpartialfromglasses.models.segmentation.unetimportUNet,UNetDecoder# vanilla Unetunet=UNet()# let's change the encoderunet=UNet.from_encoder(partial(AutoModel.from_name,'efficientnet_b1'))# mmm I want more layers in the decoder!unet=UNet(decoder=partial(UNetDecoder,widths=[256,128,64,32,16]))# maybe resnet was betterunet=UNet(encoder=lambda**kwargs:ResNet.resnet26(**kwargs).encoder)# same APIunet.summary(input_shape=(1,224,224))More examples# change the decoder partmodel=ResNet.resnet18(pretrained=True)my_head=nn.Sequential(nn.AdaptiveAvgPool2d((1,1)),nn.Flatten(),nn.Linear(model.encoder.widths[-1],512),nn.Dropout(0.2),nn.ReLU(),nn.Linear(512,1000))model.head=my_headx=torch.rand((1,3,224,224))model(x).shape#torch.Size([1, 1000])Pretrained ModelsI am currently working on the pretrained models and the best way to make them availableThis is a list of all the pretrained models available so far!. They are all trained onImageNet.I used abatch_size=64and a GTX 1080ti to evaluale the models.top1top5timebatch_sizeefficientnet_b30.820340.9603199.59964regnety_0320.819580.95964136.51864deit_small_patch16_2240.810820.95316132.86864resnet50d0.804920.9512897.582764cse_resnet500.802920.95048108.76564efficientnet_b20.801260.95124127.17764resnext101_32x8d0.79210.94556290.3864wide_resnet101_20.78910.94344277.75564wide_resnet50_20.784640.94064201.63464efficientnet_b10.78310.9409698.714364resnet1520.78250.93982186.19164regnetx_0320.77920.93996319.55864resnext50_32x4d0.776280.9368114.32564regnety_0160.776040.9370296.54764efficientnet_b00.773320.9356667.214764resnet1010.773140.93556134.14864densenet1610.771460.93602239.38864resnet34d0.771180.9341859.993864densenet2010.769320.9339158.51464regnetx_0160.766840.932891.753664resnet26d0.7660.9318870.645364regnety_0080.762380.9302654.128664resnet500.760120.9293489.797664densenet1690.756280.9281127.07764resnet260.753940.9258465.580164resnet340.750960.9224656.898564regnety_0060.750680.9247455.561164regnetx_0080.747880.9219457.955964densenet1210.744720.91974104.1364deit_tiny_patch16_2240.74370.9189866.66264vgg19_bn0.742160.91848169.35764regnety_0040.737660.9163868.489364regnetx_0060.736820.9156881.470364vgg16_bn0.734760.91536150.31764vgg190.72360.9085155.85164regnetx_0040.722980.9064458.004964vgg160.716280.90368135.39864vgg13_bn0.716180.9036129.07764vgg11_bn0.704080.8972486.945964vgg130.699840.89306116.05264regnety_0020.69980.8942246.80464resnet180.696440.8898246.202964vgg110.688720.8865879.413664regnetx_0020.686580.8824445.921164Assuming you want to loadefficientnet_b1:fromglasses.modelsimportEfficientNet,AutoModel,AutoConfig# load it using AutoModelmodel=AutoModel.from_pretrained('efficientnet_b1')# or from its own classmodel=EfficientNet.efficientnet_b1(pretrained=True)# you may also need to get the correct transformation that must be applied on the inputcfg=AutoConfig.from_name('efficientnet_b1')transform=cfg.transformINFO:root:Loaded efficientnet_b1 pretrained weights.
INFO:root:Loaded efficientnet_b1 pretrained weights.In this case,transformisCompose(
    Resize(size=240, interpolation=PIL.Image.BICUBIC)
    CenterCrop(size=(240, 240))
    ToTensor()
    Normalize(mean=(0.485, 0.456, 0.406), std=(0.229, 0.224, 0.225))
)Deep CustomizationAll models are composed by sharable parts:BlockLayerEncoderHeadDecoderBlockEach model has its building block, they are noted by*Block. In each block, all the weights are in the.blockfield. This makes it very easy to customize one specific model.fromglasses.models.classification.vggimportVGGBasicBlockfromglasses.models.classification.resnetimportResNetBasicBlock,ResNetBottleneckBlock,ResNetBasicPreActBlock,ResNetBottleneckPreActBlockfromglasses.models.classification.senetimportSENetBasicBlock,SENetBottleneckBlockfromglasses.models.classification.resnetxtimportResNetXtBottleNeckBlockfromglasses.models.classification.densenetimportDenseBottleNeckBlockfromglasses.models.classification.wide_resnetimportWideResNetBottleNeckBlockfromglasses.models.classification.efficientnetimportEfficientNetBasicBlockFor example, if we want to add Squeeze and Excitation to the resnet bottleneck block, we can justfromglasses.nn.attimportSpatialSEfromglasses.models.classification.resnetimportResNetBottleneckBlockclassSEResNetBottleneckBlock(ResNetBottleneckBlock):def__init__(self,in_features:int,out_features:int,squeeze:int=16,*args,**kwargs):super().__init__(in_features,out_features,*args,**kwargs)# all the weights are in block, we want to apply se after the weightsself.block.add_module('se',SpatialSE(out_features,reduction=squeeze))SEResNetBottleneckBlock(32,64)Then, we can use the class methods to create the new models following the existing architecture blueprint, for example, to createse_resnet50ResNet.resnet50(block=ResNetBottleneckBlock)The cool thing is each model has the same api, if I want to create a vgg13 with theResNetBottleneckBlockI can justfromglasses.modelsimportVGGmodel=VGG.vgg13(block=SEResNetBottleneckBlock)model.summary()Some specific model can require additional parameter to the block, for exampleMobileNetV2also required aexpansionparameter so ourSEResNetBottleneckBlockwon't work.LayerALayeris a collection of blocks, it is used to stack multiple blocks together following some logic. For example,ResNetLayerfromglasses.models.classification.resnetimportResNetLayerResNetLayer(64,128,depth=2)EncoderThe encoder is what encoders a vector, so the convolution layers. It has always two very important parameters.widthsdepthswidthsis the wide at each layer, so how much features there aredepthsis the depth at each layer, so how many blocks there areFor example,ResNetEncoderwill creates multipleResNetLayerbased on the len ofwidthsanddepths. Let's see some example.fromglasses.models.classification.resnetimportResNetEncoder# 3 layers, with 32,64,128 features and 1,2,3 block eachResNetEncoder(widths=[32,64,128],depths=[1,2,3])All encoders are subclass ofEncoderthat allows us to hook on specific stages to get the featuers. All you have to do is first call.featuresto notify the model you want to receive the features, and then pass an input.enc=ResNetEncoder()enc.featuresenc(torch.randn((1,3,224,224)))print([f.shapeforfinenc.features])Remembereach model has always a.decoderfieldfromglasses.modelsimportResNetmodel=ResNet.resnet18()model.encoder.widths[-1]The encoder knows the number of output features, you can access them byFeaturesEach encoder can return a list of features accessable by the.featuresfield. You need to call it once before in order to notify the encoder we wish to also store the featuresfromglasses.models.classification.resnetimportResNetEncoderx=torch.randn(1,3,224,224)enc=ResNetEncoder()enc.features# call it onceenc(x)features=enc.features# now we have all the features from each layer (stage)[print(f.shape)forfinfeatures]# torch.Size([1, 64, 112, 112])# torch.Size([1, 64, 56, 56])# torch.Size([1, 128, 28, 28])# torch.Size([1, 256, 14, 14])HeadHead is the last part of the model, it usually perform the classificationfromglasses.models.classification.resnetimportResNetHeadResNetHead(512,n_classes=1000)DecoderThe decoder takes the last feature from the.encoderand decode it. This is usually done insegmentationmodels, such as Unet.fromglasses.models.segmentation.unetimportUNetDecoderx=torch.randn(1,3,224,224)enc=ResNetEncoder()enc.features# call it oncex=enc(x)features=enc.features# we need to tell the decoder the first feature size and the size of the lateral featuresdec=UNetDecoder(start_features=enc.widths[-1],lateral_widths=enc.features_widths[::-1])out=dec(x,features[::-1])out.shapeThis object oriented structure allows to reuse most of the code across the modelsModelsThe models so far😥 = I don't have enough GPU RAMnameParametersSize (MB)resnet1811,689,51244.59resnet2615,995,17661.02resnet26d16,014,40861.09resnet3421,797,67283.15resnet34d21,816,90483.22resnet5025,557,03297.49resnet50d25,576,26497.57resnet10144,549,160169.94resnet15260,192,808229.62resnet20064,673,832246.71se_resnet1811,776,55244.92se_resnet3421,954,85683.75se_resnet5028,071,976107.09se_resnet10149,292,328188.04se_resnet15266,770,984254.71cse_resnet1811,778,59244.93cse_resnet3421,958,86883.77cse_resnet5028,088,024107.15cse_resnet10149,326,872188.17cse_resnet15266,821,848254.91resnext50_32x4d25,028,90495.48resnext101_32x8d88,791,336338.71resnext101_32x16d194,026,792740.15resnext101_32x32d468,530,4721787.30resnext101_32x48d828,411,1763160.14regnetx_0022,684,79210.24regnetx_0045,157,51219.67regnetx_0066,196,04023.64regnetx_0087,259,65627.69regnetx_0169,190,13635.06regnetx_03215,296,55258.35regnety_0023,162,99612.07regnety_0044,344,14416.57regnety_0066,055,16023.10regnety_0086,263,16823.89regnety_01611,202,43042.73regnety_03219,436,33874.14resnest14d10,611,68840.48resnest26d17,069,44865.11resnest50d27,483,240104.84resnest50d_1s4x24d25,677,00097.95resnest50d_4s2x40d30,417,592116.03resnest101e48,275,016184.15resnest200e70,201,544267.80resnest269e7,551,11228.81wide_resnet50_268,883,240262.77wide_resnet101_2126,886,696484.03densenet1217,978,85630.44densenet16914,149,48053.98densenet20120,013,92876.35densenet16128,681,000109.41fishnet9916,630,31263.44fishnet15024,960,80895.22vgg11132,863,336506.83vgg13133,047,848507.54vgg16138,357,544527.79vgg19143,667,240548.05vgg11_bn132,868,840506.85vgg13_bn133,053,736507.56vgg16_bn138,365,992527.82vgg19_bn143,678,248548.09efficientnet_b05,288,54820.17efficientnet_b17,794,18429.73efficientnet_b29,109,99434.75efficientnet_b312,233,23246.67efficientnet_b419,341,61673.78efficientnet_b530,389,784115.93efficientnet_b643,040,704164.19efficientnet_b766,347,960253.10efficientnet_b8😥😥efficientnet_l2😥😥efficientnet_lite04,652,00817.75efficientnet_lite15,416,68020.66efficientnet_lite26,092,07223.24efficientnet_lite38,197,09631.27efficientnet_lite413,006,56849.62vit_small_patch16_22448,602,344185.40vit_base_patch16_22486,415,592329.65vit_base_patch16_38486,415,592329.65vit_base_patch32_38488,185,064336.40vit_huge_patch16_224631,823,0802410.21vit_huge_patch32_384634,772,2002421.46vit_large_patch16_224304,123,8801160.14vit_large_patch16_384304,123,8801160.14vit_large_patch32_384306,483,1761169.14deit_tiny_patch16_2245,872,40022.40deit_small_patch16_22422,359,63285.30deit_base_patch16_22487,184,592332.58mobilenetv23,504,87213.37unet23,202,53088.51deit_base_patch16_38487,184,592332.58CreditsMost of the weights were trained by other people and adapted to glasses. It is worth citepytorch-image-modelstorchvision