Package: mctorch-lib
McTorch Lib, a manifold optimization library for deep learningMcTorch is a Python library that adds manifold optimization functionality toPyTorch.McTorch:Leverages tensor computation and GPU acceleration from PyTorch.Enables optimization on manifold constrained tensors to address nonlinear optimization problems.Facilitates constrained weight tensors in deep learning layers.Sections:More about McTorchUsing McTorch for OptimizationUsing McTorch for Deep LearningFunctionality SupportedInstallationRelease and ContributionTeamReferenceMore about McTorchMcTorch builds on top of PyTorch and supports all PyTorch functions in addition to Manifold optimization. This is done to ensure researchers and developers using PyTorch can easily experiment with McTorch functions. McTorch's manifold implementations and optimization methods are derived from the Matlab toolboxManoptand the Python toolboxPymanopt.Using McTorch for OptimizationInitialize Parameter- McTorch manifold parameters are same as PyTorch parameters (mctorch.nn.Parameter) and requires just addition of one property to parameter initialization to constrain the parameter values.Define Cost- Cost function can be any PyTorch function using the above parameter mixed with non constrained parameters.Optimize- Any optimizer frommctorch.optimcan be used to optimize the cost function using same functionality as any PyTorch code.PCA Exampleimporttorchimportmctorch.nnasmnnimportmctorch.optimasmoptim# Random data with high variance in first two dimensionX=torch.diag(torch.FloatTensor([3,2,1])).matmul(torch.randn(3,200))# 1. Initialize Parametermanifold_param=mnn.Parameter(manifold=mnn.Stiefel(3,2))# 2. Define Cost - squared reconstruction errordefcost(X,w):wTX=torch.matmul(w.transpose(1,0),X)wwTX=torch.matmul(w,wTX)returntorch.sum((X-wwTX)**2)# 3. Optimizeoptimizer=moptim.rAdagrad(params=[manifold_param],lr=1e-2)forepochinrange(30):cost_step=cost(X,manifold_param)print(cost_step)cost_step.backward()optimizer.step()optimizer.zero_grad()Using McTorch for Deep LearningMulti Layer Perceptron Exampleimporttorchimportmctorch.nnasmnnimporttorch.nn.functionalasF# a torch module using constrained linear layersclassManifoldMLP(nn.Module):def__init__(self):super(ManifoldMLP,self).__init__()self.layer1=mnn.rLinear(in_features=28*28,out_features=100,weight_manifold=mnn.Stiefel)self.layer2=mnn.rLinear(in_features=100,out_features=100,weight_manifold=mnn.PositiveDefinite)self.output=mnn.rLinear(in_features=100,out_features=10,weight_manifold=mnn.Stiefel)defforward(self,x):x=F.relu(self.layer1(x))x=F.relu(self.layer2(x))x=F.log_softmax(self.output(x),dim=0)returnx# create module object and compute cost by applying module on inputsmlp_module=ManifoldMLP()cost=mlp_module(inputs)More examples added -hereFunctionality SupportedThis would be an ever increasing list of features. McTorch currently supports:ManifoldsStiefelPositive DefiniteAll manifolds support k multiplier as well.OptimizersSGDAdagradConjugateGradientLayersLinearConv1d, Conv2d, Conv3dInstallationAfter installing PyTorch can be installed withpython setup.py installLinuxsourceactivatemyenv
condainstallnumpysetuptools# Add LAPACK support for the GPU if neededcondainstall-cpytorchmagma-cuda90# or [magma-cuda80 | magma-cuda92 | magma-cuda100 ] depending on your cuda versioncondainstallpytorchtorchvisioncudatoolkit=9.0-cpytorch# or cudatoolkit=10.0 | cudatoolkit=10.1 | .. depending on your cuda versionpipinstallmctorch-libRelease and ContributionMcTorch is currently under development and any contributions, suggestions and feature requests are welcome. We'd closely follow PyTorch stable versions to keep the base updated and will have our own versions for other additions.McTorch is released under the open source3-clause BSD License.TeamMayank MeghwanshiBamdev MishraPratik JawanpuriaHiroyuki KasaiAnoop KunchukuttanReferencePlease cite [1] if you found this code useful.McTorch, a manifold optimization library for deep learning[1] M. Meghawanshi, P. Jawanpuria, A. Kunchukuttan, H. Kasai, and B. Mishra,McTorch, a manifold optimization library for deep learning@techreport{meghwanshi2018mctorch,
  title={McTorch, a manifold optimization library for deep learning},
  author={Meghwanshi, Mayank and Jawanpuria, Pratik and Kunchukuttan, Anoop and Kasai, Hiroyuki and Mishra, Bamdev},
  institution={arXiv preprint arXiv:1810.01811},
  year={2018}
}