Package: hyperlight
HyperLightHypernetworks in Pytorch made easyTL;DRHyperLight is a Pytorch library designed to make implementing hypernetwork models easy and painless.
What sets HyperLight apart from other hypernetwork implementations:Bring your own architecture– Reuse your existing model code.Principled Parametrizations and Initializations– Default networks can have unstable training dynamics, HyperLight has good defaults that lead to improved training [1].Work with pretrained models– Use pretrained weights as part of the hypernetwork initialization.Seamless Composability– It's hypernets all the way down! Hypernetize hypernet models without issue.Pytorch-nicAPI design– Parameters are treated as an attribute of the layer, preventing the need for rewriting PyTorch modules.[1]Non-Proportional Parametrizations for Stable Hypernetwork LearningInstallationTo install thestableversion of HyperLight viapip:pipinstallhyperlightOr for thelatestversion:pipinstallgit+https://github.com/JJGO/hyperlight.gitFor themanualinstall:# clone itgitclonehttps://github.com/JJGO/hyperlight# install dependenciespython-mpipinstall-r./hyperlight/requirements.txt# only dependency is PyTorch# add this to your .bashrc/.zshrcexportPYTHONPATH="$PYTHONPATH:/path/to/hyperlight)"Getting StartedThe main advantage of HyperLight is that it allows to easily reuse existing networks without having to redo the model code.For example, here's a Bayesian Neural Hypernetwork for the resnet18 architecture:fromtorchvision.modelsimportresnet18importhyperlightashl# First, instantiate the main network and# hyperparametrize all convolutional weightsmainnet=resnet18()modules=hl.find_modules_of_type(mainnet,[nn.Conv2d])# Replace nn.Parameter objects with ExternalParametersmainnet=hl.hypernetize(mainnet,modules=modules)# Get the spec of the weights we need to predictparameter_shapes=mainnet.external_shapes()# We can predict these shapes any way we want,# but hyperlight provides hypernetwork modelshyperparam_shape={'h':(10,)}# 10-dim inputhypernet=hl.Hypernet(input_shapes=hyperparam_shape,output_shapes=parameter_shapes,hidden_sizes=[16,64,128],)# Now, instead of model(input) we first predict the main network weightsparameters=hl.hypernet(h=hyperpameter_input)# and then use the main networkwithmainnet.using_externals(parameters):# Within this context manager, the weights are accessibleprediction=mainnet(input)# After this point, weights are removedWe can also wrap this intonn.Moduleto pair-up the hypernet with the main network and have a nicer API:classHyperResNet18(nn.Module):def__init__(self,hypernet_layers:List[]):super().__init__()mainnet=resnet18()modules=hl.find_modules_of_type(mainnet,[nn.Conv2d])self.mainnet=hl.hypernetize(mainnet,modules=modules)self.hypernet=hl.Hypernet(input_shapes={'h':(10,)},output_shapes=parameter_shapes,layer_sizes=[16,64,128],)defforward(self,input,hyper_input):parameters=self.hypernet(h=hyper_input)withself.mainnet.using_externals(parameters):prediction=self.mainnet(input)returninputWith HyperLight, we can reuse the pretrained weights by setting them as independent weights:classHyperResNet18(nn.Module):def__init__(self,hypernet_layers:List[]):super().__init__()# Load pretrained weightsmainnet=resnet18(pretrained=True)modules=hl.find_modules_of_type(mainnet,[nn.Conv2d])self.mainnet,weights=hl.hypernetize(mainnet,modules=modules,return_values=True)# Construct from existingself.hypernet=hl.Hypernet.from_existing(weights,# weights encode shape and initializationinput_shapes={'h':(10,)},output_shapes=parameter_shapes,layer_sizes=[16,64,128],)defforward(self,input,hyper_input):parameters=self.hypernet(h=hyper_input)withself.mainnet.using_externals(parameters):prediction=self.mainnet(input)returninputTutorialConceptsHyperLight introduces a few new concepts:HyperModule– A specializednn.Moduleobject that can hold both regular parameters
andExternalParametersto be predicted by an external hypernetwork.ExternalParameter–nn.Parameterreplacement that only stores the required shape of the
externalized parameter. Parameter data can be set and reset with the hypernetwork predictions.HyperNetwork–nn.Modulethat predicts a main network parameters for a given input.Defining aHyperModulewithExternalParametersHere is an example of how we define a hypernetized Linear layer. We need to make sure to
define theExternalParameterproperties with their correct shapes.importtorch.nn.functionalasFimporthyperlightashlclassHyperLinear(hl.HyperModule):"""Implementation of a nn.Linear layer but with external parametersthat will be predicted by an external hypernetwork"""in_features:intout_features:intdef__init__(self,in_features:int,out_features:int,bias:bool=True)->None:super().__init__()assertisinstance(in_features,int)andisinstance(out_features,int)self.in_features=in_featuresself.out_features=out_featuresself.weight=hl.ExternalParameter(shape=(out_features,in_features))ifbias:self.bias=hl.ExternalParameter(shape=(out_features,))else:self.bias=Nonedefforward(self,input:Tensor)->Tensor:returnF.linear(input,self.weight,self.bias)Once defined, we can make use of this module as follows:>>>layer=HyperLinear(in_features=8,out_features=16)>>>layer.external_shapes(){'weight':(16,8),'bias':(16,)}>>>x=torch.zeros(1,8)# We need to set the weights before using the layer>>>layer(x)[...]AttributeError:UninitializedExternalParameter,pleasesetthevaluefirst# Initialize the external weights>>>layer.set_externals(weight=torch.rand(size=(16,8)),bias=torch.zeros((16,)))>>>layer(x).shapetorch.Size([1,16])# Once we are done, we reset the external parameter values>>>layer.reset_externals()Alternatively, we can use theusing_externalscontextmanager that will set and reset
the parameters accordingly:params(weight=torch.rand(size=(16,8)),bias=torch.zeros((16,)))withlayer.using_externals(params):y=layer(x)Dynamically hypernetizing modulesHyperLight supportsdynamicHyperModule creation using thehypernetizehelper.
We need to specify which parameters we want to remove from the module and convert toExternalParameterobjects:>>>fromtorchimportnn>>>importhyperlightashl>>>layer=nn.Linear(in_features=8,out_features=16)>>>layer=hl.hypernetize(layer,parameters=[layer.weight,layer.bias])>>>layerHypernetizedLinear()>>>layer.external_shapes(){'weight':(16,8),'bias':(16,)}hypernetizeis recursive, and supports entire modules being specified:>>>model=nn.Sequential(OrderedDict({'conv':nn.Conv2d(3,128,3),'norm':nn.BatchNorm2d(128),'relu':nn.ReLU(),'pool':nn.AdaptiveAvgPool2d((1,1)),'out':nn.Linear(128,10)}))>>>model=hl.hypernetize(model,modules=[model.conv,model.out])>>>model.external_shapes(){'conv.weight':(128,3,3,3),'conv.bias':(128,),'out.weight':(10,128),'out.bias':(10,)}Finding modules and parametersIn addition, HyperLight provides several routines to recursively search for parameters and modules to feed intohypernetize:find_modules_of_type(model, module_types)– Find modules of a certain type,
e.g.nn.Linearornn.Conv2dfind_modules_from_patterns(model, globs=None, regex=None)– Find modules that match
specific patterns using globs, e.g.*.conv; or regexes, e.g.layer[1-3].*convfind_parameters_from_patterns(model, globs=None, regex=None)– Find parameters
that match specific patterns.Some examples on a ResNet18 architecture:>>>fromtorchvision.modelsimportresnet18>>>importhyperlightashl>>>model=resnet18()# Find all convolutions>>>hl.find_modules_of_type(model,[nn.Conv2d]){'conv1':Conv2d(3,64,kernel_size=(7,7),stride=(2,2),padding=(3,3),bias=False),'layer1.0.conv1':Conv2d(64,64,kernel_size=(3,3),stride=(1,1),padding=(1,1),bias=False),'layer1.0.conv2':Conv2d(64,64,kernel_size=(3,3),stride=(1,1),padding=(1,1),bias=False),...# Find the first convolution of each ResNet block>>>hl.find_modules_from_patterns(model,regex=['^layer\d.0.conv1']){'layer1.0.conv1':Conv2d(64,64,kernel_size=(3,3),stride=(1,1),padding=(1,1),bias=False),'layer2.0.conv1':Conv2d(64,128,kernel_size=(3,3),stride=(2,2),padding=(1,1),bias=False),'layer3.0.conv1':Conv2d(128,256,kernel_size=(3,3),stride=(2,2),padding=(1,1),bias=False),'layer4.0.conv1':Conv2d(256,512,kernel_size=(3,3),stride=(2,2),padding=(1,1),bias=False)}# Get only the convolutional weights of the first block (no biases)>>>hl.find_parameters_from_patterns(model,globs=['layer1*conv*.weight']).keys()dict_keys(['layer1.0.conv2.weight','layer1.0.conv1.weight','layer1.1.conv1.weight','layer1.1.conv2.weight'])Other methodsHyperLight goes beyond hypernetworks and helps implement other Deep Learning techniques related to hypernetworks.As an example, the following code implementsFiLM. Instead of having to modify
our entire forward pass to keep track of the $\gamma$ and $\beta$ coefficients, we can have HyperLight handle that for us:importhyperlightashlclassFiLM(hl.HyperModule):def__init__(self,n_features:int):super().__init__()self.n_features=n_featuresself.gamma=hl.ExternalParameter((n_features,))self.beta=hl.ExternalParameter((n_features,))defforward(self,x):returnself.gamma*x+self.betaclassCNNwithFiLM(hl.HyperModule):def__init__(self):super().__init__()self.layers=nn.Sequential([nn.Conv2d(3,64,kernel_size=3,padding=1),FiLM(64),nn.LeakyReLU(),nn.BatchNorm2d(64)nn.Conv2d(64,128,kernel_size=3,padding=1),FiLM(128),nn.LeakyReLU(),nn.BatchNorm2d(128)nn.Conv2d(128,256,kernel_size=3,padding=1),FiLM(256),nn.LeakyReLU(),nn.BatchNorm2d(256)nn.AdaptiveAvgPool2d((1,1)),])defFiLM_Model(nn.Module):def__init__(self,embedding_size):self.main=CNNwithFiLM()self.aux=hl.Hypernet(input_shapes={'film_input':(embedding_size,)},output_shapes=self.main.external_shapes(),hidden_sizes=[],)defforward(self,input,conditioning):params=self.aux(film_input=conditioning)withself.main.using_externals(params):returnself.main(input)CitationIf you find our work or any of our materials useful, please cite our paper:@article{ortiz2023nonproportional,
  title={Non-Proportional Parametrizations for Stable Hypernetwork Learning},
  author={Jose Javier Gonzalez Ortiz and John Guttag and Adrian Dalca},
  year={2023},
  journal={arXiv:2304.07645},
}