Package: torchlayers-nightly
VersionDocsTestsCoverageStylePyPIPythonPyTorchDockertorchlayersis a library based onPyTorchprovidingautomatic shape and dimensionality inference oftorch.nnlayers+ additional
building blocks featured in current SOTA architectures (e.g.Efficient-Net).Above requires no user intervention (except single call totorchlayers.build)
similarly to the one seen inKeras.Main functionalities:Shape inferencefor most oftorch.nnmodule (convolutional, recurrent, transformer, attention and linear layers)Dimensionality inference(e.g.torchlayers.Convworking astorch.nn.Conv1d/2d/3dbased oninput shape)Shape inference of custom modules(see examples section)AdditionalKeras-likelayers(e.g.torchlayers.Reshapeortorchlayers.StandardNormalNoise)Additional SOTA layersmostly from ImageNet competitions
(e.g.PolyNet,Squeeze-And-Excitation,StochasticDepth)Useful defaults("same"padding and defaultkernel_size=3forConv, dropout rates etc.)Zero overhead andtorchscriptsupportKeep in mind this library works almost exactly like PyTorch originally.
What that means is you can useSequential,define your own networks of any complexity usingtorch.nn.Module, create new layers with shape inference etc.See below to get some intuition about library.ExamplesFor full functionality please checktorchlayers documentation.
Below examples should introduce all necessary concepts you should know.Basic classifierAlltorch.nnmodules can be used throughtorchlayersandeach module with input shapewill be appropriately modified with it's input inferable counterpart.importtorchlayersastlclassClassifier(tl.Module):def__init__(self):super().__init__()self.conv1=tl.Conv2d(64,kernel_size=6)self.conv2=tl.Conv2d(128,kernel_size=3)self.conv3=tl.Conv2d(256,kernel_size=3,padding=1)# New layer, more on that in the next exampleself.pooling=tl.GlobalMaxPool()self.dense=tl.Linear(10)defforward(self,x):x=torch.relu(self.conv1(x))x=torch.relu(self.conv2(x))x=torch.relu(self.conv3(x))returnself.dense(self.pooling(x))# Pass model and any example inputs afterwardsclf=tl.build(Classifier(),torch.randn(1,3,32,32))Abovetorchlayers.Linear(out_features=10)is used. It is "equivalent" to
original PyTorch'storch.nn.Linear(in_features=?, out_features=10)wherein_featureswill be inferred from example input input duringtorchlayers.buildcall.Same thing happens withtorch.nn.Conv2d(in_channels, out_channels, kernel_size, ...)which can be replaced directly bytl.Conv2d(out_channels, kernel_size, ...).Just remember to pass example input through the network!Simple image and text classifier in one!We will use single "model" for both tasks.
Firstly let's define it usingtorch.nnandtorchlayers:importtorchimporttorchlayersastl# torch.nn and torchlayers can be mixed easilymodel=torch.nn.Sequential(tl.Conv(64),# specify ONLY out_channelstorch.nn.ReLU(),# use torch.nn wherever you wishtl.BatchNorm(),# BatchNormNd inferred from inputtl.Conv(128),# Default kernel_size equal to 3tl.ReLU(),tl.Conv(256,kernel_size=11),# "same" padding as defaulttl.GlobalMaxPool(),# Known from Kerastl.Linear(10),# Output for 10 classes)print(model)Above would give you model's summary like this (notice question marks for not yet inferred values):Sequential((0):Conv(in_channels=?,out_channels=64,kernel_size=3,stride=1,padding=same,dilation=1,groups=1,bias=True,padding_mode=zeros)(1):ReLU()(2):BatchNorm(num_features=?,eps=1e-05,momentum=0.1,affine=True,track_running_stats=True)(3):Conv(in_channels=?,out_channels=128,kernel_size=3,stride=1,padding=same,dilation=1,groups=1,bias=True,padding_mode=zeros)(4):ReLU()(5):Conv(in_channels=?,out_channels=256,kernel_size=11,stride=1,padding=same,dilation=1,groups=1,bias=True,padding_mode=zeros)(6):GlobalMaxPool()(7):Linear(in_features=?,out_features=10,bias=True))Now you canbuild/instantiate your model with example input (in this case MNIST-like):mnist_model=tl.build(model,torch.randn(1,3,28,28))Or if it's text classification you are after, same model could be built with differentinput shape(e.g. for text classification using300dimensional pretrained embedding):# [batch, embedding, timesteps], first dimension > 1 for BatchNorm1d to worktext_model=tl.build(model,torch.randn(2,300,1))Finally, you canprintboth models after instantiation, provided below side
by-side for readability (notice different dimenstionality, e.g.Conv2dvsConv1daftertorchlayers.build):# TEXT CLASSIFIER                 MNIST CLASSIFIERSequential(Sequential((0):Conv1d(300,64)(0):Conv2d(3,64)(1):ReLU()(1):ReLU()(2):BatchNorm1d(64)(2):BatchNorm2d(64)(3):Conv1d(64,128)(3):Conv2d(64,128)(4):ReLU()(4):ReLU()(5):Conv1d(128,256)(5):Conv2d(128,256)(6):GlobalMaxPool()(6):GlobalMaxPool()(7):Linear(256,10)(7):Linear(256,10)))As you can see both modules "compiled" into originalpytorchlayers.Custom modules with shape inference capabilitiesUser can define any module and make it shape inferable withtorchlayers.inferfunction:# Class defined with in_features# It might be a good practice to use _ prefix and Impl as postfix# to differentiate from shape inferable versionclass_MyLinearImpl(torch.nn.Module):def__init__(self,in_features:int,out_features:int):super().__init__()self.weight=torch.nn.Parameter(torch.randn(out_features,in_features))self.bias=torch.nn.Parameter(torch.randn(out_features))defforward(self,inputs):returntorch.nn.functional.linear(inputs,self.weight,self.bias)MyLinear=tl.infer(_MyLinearImpl)# Build and use just like any other layer in this librarylayer=tl.build(MyLinear(out_features=32),torch.randn(1,64))layer(torch.randn(1,64))By defaultinputs.shape[1]will be used asin_featuresvalue
during initialforwardpass. If you wish to use differentindex(e.g. to infer usinginputs.shape[3]) useMyLayer = tl.infer(_MyLayerImpl, index=3)as a decorator.Autoencoder with inverted residual bottleneck and pixel shufflePlease check code comments anddocumentationif needed. If you are unsure what autoencoder is you could seethis example blog post.Below is a convolutional denoising autoencoder example forImageNet-like images.
Think of it like a demonstration of capabilities of different layers
and building blocks provided bytorchlayers.# Input - 3 x 256 x 256 for ImageNet reconstructionclassAutoEncoder(torch.nn.Module):def__init__(self):super().__init__()self.encoder=tl.Sequential(tl.StandardNormalNoise(),# Apply noise to input imagestl.Conv(64,kernel_size=7),tl.activations.Swish(),# Direct access to module .activationstl.InvertedResidualBottleneck(squeeze_excitation=False),tl.AvgPool(),# shape 64 x 128 x 128, kernel_size=2 by defaulttl.HardSwish(),# Access simply through tltl.SeparableConv(128),# Up number of channels to 128tl.InvertedResidualBottleneck(),# Default with squeeze excitationtorch.nn.ReLU(),tl.AvgPool(),# shape 128 x 64 x 64, kernel_size=2 by defaulttl.DepthwiseConv(256),# DepthwiseConv easier to use# Pass input thrice through the same weights like in PolyNettl.Poly(tl.InvertedResidualBottleneck(),order=3),tl.ReLU(),# all torch.nn can be accessed via tltl.MaxPool(),# shape 256 x 32 x 32tl.Fire(out_channels=512),# shape 512 x 32 x 32tl.SqueezeExcitation(hidden=64),tl.InvertedResidualBottleneck(),tl.MaxPool(),# shape 512 x 16 x 16tl.InvertedResidualBottleneck(squeeze_excitation=False),# Randomly switch off the last two layers with 0.5 probabilitytl.StochasticDepth(torch.nn.Sequential(tl.InvertedResidualBottleneck(squeeze_excitation=False),tl.InvertedResidualBottleneck(squeeze_excitation=False),),p=0.5,),tl.AvgPool(),# shape 512 x 8 x 8)# This one is more "standard"self.decoder=tl.Sequential(tl.Poly(tl.InvertedResidualBottleneck(),order=2),# Has ICNR initialization by default after calling `build`tl.ConvPixelShuffle(out_channels=512,upscale_factor=2),# Shape 512 x 16 x 16 after PixelShuffletl.Poly(tl.InvertedResidualBottleneck(),order=3),tl.ConvPixelShuffle(out_channels=256,upscale_factor=2),# Shape 256 x 32 x 32tl.Poly(tl.InvertedResidualBottleneck(),order=3),tl.ConvPixelShuffle(out_channels=128,upscale_factor=2),# Shape 128 x 64 x 64tl.Poly(tl.InvertedResidualBottleneck(),order=4),tl.ConvPixelShuffle(out_channels=64,upscale_factor=2),# Shape 64 x 128 x 128tl.InvertedResidualBottleneck(),tl.Conv(256),tl.Dropout(),# Defaults to 0.5 and Dropout2d for imagestl.Swish(),tl.InstanceNorm(),tl.ConvPixelShuffle(out_channels=32,upscale_factor=2),# Shape 32 x 256 x 256tl.Conv(16),tl.Swish(),tl.Conv(3),# Shape 3 x 256 x 256)defforward(self,inputs):returnself.decoder(self.encoder(inputs))Now one can instantiate the module and use it withtorch.nn.MSELossas per usual.autoencoder=tl.build(AutoEncoder(),torch.randn(1,3,256,256))InstallationpipLatest release:pipinstall--usertorchlayersNightly:pipinstall--usertorchlayers-nightlyDockerCPU standaloneand various versions ofGPU enabledimages are available
atdockerhub.For CPU quickstart, issue:dockerpullszymonmaszke/torchlayers:18.04Nightly builds are also available, just prefix tag withnightly_. If you are going forGPUimage make sure you havenvidia/dockerinstalled and it's runtime set.ContributingIf you find issue or would like to see some functionality (or implement one), pleaseopen new Issueorcreate Pull Request.