How to use a custom method for prediction using PyTorch JIT in a custom handler for production environment

username: mhannani
cooked: <p>I want to deploy a model (local at the first step) using Torchserve.</p>
<p>I trained my model, and saved it in eager mode</p>
<p>Model architecture:</p>
<p><strong>- Encoder</strong></p>
<p>Simple CNN model using the pretrained Inception v3.</p>
<p><strong>- Decoder</strong><br>
A langauge model to generate caption(using LSTM cell) of an image.</p>
<p><strong>- Encoder_to_Decoder</strong><br>
hocking up the two preceding models, contains two methods, the ordinary forward and caption_image for inference.</p>
<p>WHAT I WANT: I want to execute the caption_image method when inferencing using the given ‘tensorized’ when deploying into production.</p>
<p><strong>Custom Handler</strong> (filename: custom_handler.py)</p>
<pre><code>import json
import torch
import io
from PIL import Image
from ts.torch_handler.base_handler import BaseHandler
from torchvision import transforms


class CaptionHandler(BaseHandler):
    """
    A custom model handler implementation.
    The Handler takes a image as Tensor preprocess it
    and fed it to the network for inference.
    Then got a generated caption post-process it
    and sent it as a request response.
    """

    def __init__(self):
        super().__init__()
        self.initialized = False
        self.transform = transforms.Compose([
            transforms.Resize(256),
            transforms.CenterCrop(224),
            transforms.ToTensor(),
            transforms.Normalize(mean=[0.485, 0.456, 0.406],
                                 std=[0.229, 0.224, 0.225])
        ])
        with open("index_to_name.json") as f:
            self.vocabulary = json.load(f)

    def preprocess(self, request):
        request = request[0]
        image = request.get("data")
        if image is None:
            image = request.get("body")

        image = Image.open(io.BytesIO(image))
        transformed_image = self.transform(image)

        return transformed_image

    def inference(self, data):
        model_output = self.model(data) # get stuck here !
        return model_output

    def postprocess(self, data):
        return data

    def handle(self, data, context):
        model_input = self.preprocess(data)
        # print("model input", model_input)
        model_output = self.inference(model_input)
        return self.postprocess(model_output)
</code></pre>
<p>And use that custom handler(filename: caption_handler.py):</p>
<pre><code>from custom_handler import CaptionHandler

_service = CaptionHandler()


def handle(data, context):
    """
    Handle inference requests to torchserve.
    :param data: supplied data
    :param context: the context of server
    :return: predicted caption
    """

    # if the handler not yet initialized with the context
    if not _service.initialized:
        _service.initialize(context)

    # Not data supplied
    if data is None:
        return None

    # preprocess, inference and postprocess our image
    data = _service.preprocess(data)
    data = _service.inference(data)
    data = _service.postprocess(data)

    return data
</code></pre>
<p>Then I packaged all my model artifacts using <code>torch-model-archiver</code> with this command:</p>
<pre><code>torch-model-archiver --model-name caption --version 1.0 --serialized-file checkpoints/caption.pt --extra-files ./custom_handler.py,./index_to_name.json --handler caption_handler.py --export-path model_store --force
</code></pre>
<p>In the file logs/model_log.log I got:</p>
<pre><code>MODEL_LOG - RuntimeError: forward() is missing value for argument 'captions'. Declaration: forward(__torch__.utils.models.Captioner self, Tensor images, Tensor captions) -&gt; (Tensor)
</code></pre>
<p>Indeed, in the forward method in my model class, images and captions parameters are required to train the model, but while ‘inferencing’ I want to use the image_captioner method instead to make predictions and supply only the image as a parameter to my POST request.</p>
<pre><code>curl http://127.0.0.1:8080/predictions/caption -T Data/Images/test/3637013_c675de7705.jpg
</code></pre>
<p>output:</p>
<pre><code>{
  "code": 503,
  "type": "InternalServerException",
  "message": "Prediction failed"
}
</code></pre>
<p>Thank you,</p>
<p>ANY HELP WE’LL APPRECIATED GUYS.</p>

username: marksaroufim
cooked: <p>Could you try filling <code>captions</code> with a dummy value? and then in your <code>forward()</code> function if that dummy variable is found make an inference while ignoring it</p>