Serving Model Trained with PyTorch

abc: Tensorflow has Tensorflow Serving. I know pytorch is a framework in its early stages, but how do people serve models trained with pytorch. Must it be from Python? I’m specifically looking to serve from C++.

apaszke: We don’t have a way to serve models from C++ right now, and it’s not a priority for us at this stage. There are many things like distributed training and double backward that we’ll be implementing first. Sorry!

abc: Would you say that pytorch was built with serving in mind, e.g. for an API, or more for research purposes?

apaszke: We’re more research oriented. We’re rather thinking of creating tools to export models to frameworks that are more focused on production usage like Caffe2 and TensorFlow.

abc: Also you mentioned double backward. This is the first I’ve heard of it. I found a paper by Yann LeCun on double backpropagation, but was wondering whether it’s common to use such a method.

lantiga: Hi, I’m playing with a possible solution for serving from C based on TH and THNN. It’ll be limited to statically compilable graphs of course. I should have something to share in the not so distant future.

apaszke: 

lantiga: 

apaszke: Sure that sounds cool. It doesn’t seem hacky, it’s just a graph compiler. It’s a very good start, and will likely be capable of producing small binaries. Let us know when there’s going to be any progress or in case you have any trouble. We’ll definitely showcase your solution somewhere.

Eugenio_Culurciello: 

lantiga: Making progress. As soon as I get the first MNIST example to compile I’ll share what I have.

mvitez: We need to deploy pytorch models to e.g. Android, so we need a method to export a model. This is my starting point. Can you please tell me if I am on the right way or if I am doing something totally stupid? <SCODE>import sys
import torch
from torch import nn
from torchvision import models
from torch.utils.serialization import load_lua

def dump(f):
	s = str(f.__class__)
	sys.stdout.write(s[s.rfind('.')+1:-2]+'(')
	for fa in f.previous_functions:
		if isinstance(fa[0], torch.autograd.Function):
			dump(fa[0])
			sys.stdout.write(',')
		if isinstance(fa[0], torch.nn.parameter.Parameter):
			sys.stdout.write('param,')
		elif isinstance(fa[0], torch.autograd.Variable):
			sys.stdout.write('input,')
	sys.stdout.write(')')


class MyNet(nn.Module):
    def __init__(self):
        super(MyNet, self).__init__()
        self.conv1 = nn.Conv2d(3, 16, kernel_size=1, bias=False)
        self.bn1 = nn.BatchNorm2d(16)
        self.conv2 = nn.Conv2d(3, 16, kernel_size=1, bias=True)

    def forward(self, x):
        return self.bn1(self.conv1(x))+self.conv2(x)
        

#net = models.alexnet()
#net=load_lua('model.net') #Legacy networks won't work (no support for Variables)
net = MyNet()
input=torch.autograd.Variable(torch.zeros(1,3,128,128))
output=net.forward(input)
dump(output.creator)
print('')
<ECODE> The output for the simple MyNet will be <SCODE>Add(BatchNorm(ConvNd(input,param,),param,param,),ConvNd(input,param,param,),)
<ECODE> Thanks

apaszke: 

mvitez: Yes, please. I have just sent an invitation request for slack to soumith.

lantiga: I’m not sure how profoundly things will have to be reworked with the upcoming changes in autograd, but it’s fun anyway.

lantiga: 

smth: 

mvitez: Great, very nice work, thank you.

apaszke: 

Eugenio_Culurciello: