Package: torchlayers
VersionDocsTestsCoverageStylePyPIPythonPyTorchDockertorchlayersis a library based onPyTorchprovidingautomatic shape and dimensionality inference oftorch.nnlayers+ additional
building blocks featured in current SOTA architectures (e.g.Efficient-Net).Above requires no user intervention (except single call totorchlayers.build)
similarly to the one seen inKeras.Main functionalities:Shape inferencefor most oftorch.nnmodule (convolutional, recurrent, transformer, attention and linear layers)Dimensionality inference(e.g.torchlayers.Convworking astorch.nn.Conv1d/2d/3dbased oninput shape)Shape inference of custom modules(see examples section)AdditionalKeras-likelayers(e.g.torchlayers.Reshapeortorchlayers.StandardNormalNoise)Additional SOTA layersmostly from ImageNet competitions
(e.g.PolyNet,Squeeze-And-Excitation,StochasticDepth)Useful defaults("same"padding and defaultkernel_size=3forConv, dropout rates etc.)Zero overhead andtorchscriptsupportExamplesFor full functionality please checktorchlayers documentation.
Below examples should introduce all necessary concepts you should know.Simple convolutional image and text classifierWe will use single "model" for both tasks.
Firstly let's define it usingtorch.nnandtorchlayers:importtorchimporttorchlayers# torch.nn and torchlayers can be mixed easilymodel=torch.nn.Sequential(torchlayers.Conv(64),# specify ONLY out_channelstorch.nn.ReLU(),# use torch.nn wherever you wishtorchlayers.BatchNorm(),# BatchNormNd inferred from inputtorchlayers.Conv(128),# Default kernel_size equal to 3torchlayers.ReLU(),torchlayers.Conv(256,kernel_size=11),# "same" padding as defaulttorchlayers.GlobalMaxPool(),# Known from Kerastorchlayers.Linear(10),# Output for 10 classes)print(model)Above would give you model's summary like this (notice question marks for not yet inferred values):Sequential((0):Conv(in_channels=?,out_channels=64,kernel_size=3,stride=1,padding=same,dilation=1,groups=1,bias=True,padding_mode=zeros)(1):ReLU()(2):BatchNorm(num_features=?,eps=1e-05,momentum=0.1,affine=True,track_running_stats=True)(3):Conv(in_channels=?,out_channels=128,kernel_size=3,stride=1,padding=same,dilation=1,groups=1,bias=True,padding_mode=zeros)(4):ReLU()(5):Conv(in_channels=?,out_channels=256,kernel_size=11,stride=1,padding=same,dilation=1,groups=1,bias=True,padding_mode=zeros)(6):GlobalMaxPool()(7):Linear(in_features=?,out_features=10,bias=True))Now you canbuild/instantiate your model with example input (in this case MNIST-like):mnist_model=torchlayers.build(model,torch.randn(1,3,28,28))Or if it's text classification you are after, same model could be built with differentinput shape(e.g. for text classification using300dimensional pretrained embedding):# [batch, embedding, timesteps], first dimension > 1 for BatchNorm1d to worktext_model=torchlayers.build(model,torch.randn(2,300,1))Finally, you canprintboth models after instantiation, provided below side
by-side for readability (notice different dimenstionality, e.g.Conv2dvsConv1daftertorchlayers.build):# MNIST CLASSIFIER                TEXT CLASSIFIERSequential(Sequential((0):Conv1d(300,64)(0):Conv2d(3,64)(1):ReLU()(1):ReLU()(2):BatchNorm1d(64)(2):BatchNorm2d(64)(3):Conv1d(64,128)(3):Conv2d(64,128)(4):ReLU()(4):ReLU()(5):Conv1d(128,256)(5):Conv2d(128,256)(6):GlobalMaxPool()(6):GlobalMaxPool()(7):Linear(256,10)(7):Linear(256,10)))As you can see both modules "compiled" into originalpytorchlayers.Custom modules with shape inference capabilitiesUser can define any module and make it shape inferable withtorchlayers.inferfunction:# Class defined with in_features# It might be a good practice to use _ prefix and Impl as postfix# to differentiate from shape inferable versionclass_MyLinearImpl(torch.nn.Module):def__init__(self,in_features:int,out_features:int):super().__init__()self.weight=torch.nn.Parameter(torch.randn(out_features,in_features))self.bias=torch.nn.Parameter(torch.randn(out_features))defforward(self,inputs):returntorch.nn.functional.linear(inputs,self.weight,self.bias)MyLinear=torchlayers.infer(_MyLinearImpl)# Build and use just like any other layer in this librarylayer=torchlayers.build(MyLinear(out_features=32),torch.randn(1,64))layer(torch.randn(1,64))By defaultinputs.shape[1]will be used asin_featuresvalue
during initialforwardpass. If you wish to use differentindex(e.g. to infer usinginputs.shape[3]) useMyLayer = torchlayers.infer(_MyLayerImpl, index=3)as a decorator.Autoencoder with inverted residual bottleneck and pixel shufflePlease check code comments anddocumentationif needed. If you are unsure what autoencoder is you could seethis example blog post.Below is a convolutional denoising autoencoder example forImageNet-like images.
Think of it like a demonstration of capabilities of different layers
and building blocks provided bytorchlayers.# Input - 3 x 256 x 256 for ImageNet reconstructionclassAutoEncoder(torch.nn.Module):def__init__(self):super().__init__()self.encoder=torchlayers.Sequential(torchlayers.StandardNormalNoise(),# Apply noise to input imagestorchlayers.Conv(64,kernel_size=7),torchlayers.activations.Swish(),# Direct access to module .activationstorchlayers.InvertedResidualBottleneck(squeeze_excitation=False),torchlayers.AvgPool(),# shape 64 x 128 x 128, kernel_size=2 by defaulttorchlayers.HardSwish(),# Access simply through torchlayerstorchlayers.SeparableConv(128),# Up number of channels to 128torchlayers.InvertedResidualBottleneck(),# Default with squeeze excitationtorch.nn.ReLU(),torchlayers.AvgPool(),# shape 128 x 64 x 64, kernel_size=2 by defaulttorchlayers.DepthwiseConv(256),# DepthwiseConv easier to use# Pass input thrice through the same weights like in PolyNettorchlayers.Poly(torchlayers.InvertedResidualBottleneck(),order=3),torchlayers.ReLU(),# all torch.nn can be accessed via torchlayerstorchlayers.MaxPool(),# shape 256 x 32 x 32torchlayers.Fire(out_channels=512),# shape 512 x 32 x 32torchlayers.SqueezeExcitation(hidden=64),torchlayers.InvertedResidualBottleneck(),torchlayers.MaxPool(),# shape 512 x 16 x 16torchlayers.InvertedResidualBottleneck(squeeze_excitation=False),# Randomly switch off the last two layers with 0.5 probabilitytorchlayers.StochasticDepth(torch.nn.Sequential(torchlayers.InvertedResidualBottleneck(squeeze_excitation=False),torchlayers.InvertedResidualBottleneck(squeeze_excitation=False),),p=0.5,),torchlayers.AvgPool(),# shape 512 x 8 x 8)# This one is more "standard"self.decoder=torchlayers.Sequential(torchlayers.Poly(torchlayers.InvertedResidualBottleneck(),order=2),# Has ICNR initialization by default after calling `build`torchlayers.ConvPixelShuffle(out_channels=512,upscale_factor=2),# Shape 512 x 16 x 16 after PixelShuffletorchlayers.Poly(torchlayers.InvertedResidualBottleneck(),order=3),torchlayers.ConvPixelShuffle(out_channels=256,upscale_factor=2),# Shape 256 x 32 x 32torchlayers.Poly(torchlayers.InvertedResidualBottleneck(),order=3),torchlayers.ConvPixelShuffle(out_channels=128,upscale_factor=2),# Shape 128 x 64 x 64torchlayers.Poly(torchlayers.InvertedResidualBottleneck(),order=4),torchlayers.ConvPixelShuffle(out_channels=64,upscale_factor=2),# Shape 64 x 128 x 128torchlayers.InvertedResidualBottleneck(),torchlayers.Conv(256),torchlayers.Dropout(),# Defaults to 0.5 and Dropout2d for imagestorchlayers.Swish(),torchlayers.InstanceNorm(),torchlayers.ConvPixelShuffle(out_channels=32,upscale_factor=2),# Shape 32 x 256 x 256torchlayers.Conv(16),torchlayers.Swish(),torchlayers.Conv(3),# Shape 3 x 256 x 256)defforward(self,inputs):returnself.decoder(self.encoder(inputs))Now one can instantiate the module and use it withtorch.nn.MSELossas per usual.autoencoder=torchlayers.build(AutoEncoder(),torch.randn(1,3,256,256))InstallationpipLatest release:pipinstall--usertorchlayersNightly:pipinstall--usertorchlayers-nightlyDockerCPU standaloneand various versions ofGPU enabledimages are available
atdockerhub.For CPU quickstart, issue:dockerpullszymonmaszke/torchlayers:18.04Nightly builds are also available, just prefix tag withnightly_. If you are going forGPUimage make sure you havenvidia/dockerinstalled and it's runtime set.ContributingIf you find issue or would like to see some functionality (or implement one), pleaseopen new Issueorcreate Pull Request.