Custom handler - Invalid Return type <class 'MyClass'>

dhx: <p>This is running on a model server and when I submit a request I get:</p>
<pre><code class="lang-auto">[INFO ] W-9000-model_v1_1.0-stdout MODEL_LOG - model: model_v1, Invalid return type: &lt;class 'MyClass'&gt;.
</code></pre>
<p>FWIW - it also occurs if the return type of the inference method in the custom handler is as simple as a string.</p>
<p>I have a custom_handler with a basic inference method like:</p>
<pre><code class="lang-auto">    def inference(self, ds, *args, **kwargs):
        # do stuff to data
        with torch.inference_mode():
            self.model.eval()
            pred = self.model.forward(inp).squeeze()
        

        return pred
</code></pre>

marksaroufim: <p>What are you trying to do? Typically inference function should return a list of strings or list of tensors etc…</p>
<p>FWIW you can workaround this limitation by instead doing <code>return [my_pred]</code></p>

dhx: <p>I actually updated the code. I know that <code>pred</code> is of type Tensor now.</p>
<p>How should I return this so that it works?</p>
<p>If I just do: <code>return pred</code> then I see: <code> Invalid return type: &lt;class 'torch.Tensor'&gt;.</code></p>
<p>if i do <code>return [pred]</code> then I get:</p>
<pre><code class="lang-auto">Traceback (most recent call last):
  File ".../site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "rest_inf.py", line 18, in &lt;module&gt;
    print("Inference result: ", response.json())
  File ".../site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

</code></pre>
<p>not sure what is expected here?</p>
<p>do i have to convert the tensor to a json or to some array?</p>
<p>Thanks a lot for your help <a class="mention" href="/u/marksaroufim">@marksaroufim</a></p>

marksaroufim: <p>Hmm, could you please open an issue on github and oncall can help take a look? <a href="https://github.com/pytorch/serve/issues/" class="inline-onebox">Issues · pytorch/serve · GitHub</a></p>
<p>Which version of torchserve are you using? Looking at the stacktrace I don’t see any torchserve code. A user reported a similar issue that was due to a flaky internet connection here <a href="https://github.com/pytorch/serve/issues/1134" class="inline-onebox">Failed to build Docker image in production environment · Issue #1134 · pytorch/serve · GitHub</a></p>