Debugging a Custom Handler

dhx: <p>I am very new to Torch Serving so please ignore my beginner ignorance.</p>
<p>I am trying to test out a CustomHandler, which in the <code>initialize</code> method receives a Context object.</p>
<p>The <a href="https://github.com/pytorch/serve/blob/master/ts/context.py" rel="noopener nofollow ugc">context</a> file needs to be provided into the Initialize function.</p>
<p>Now, I am able to create a .mar file using the <code>torch-model-archiver</code>, and also deploy it to a URL using the <code>torchserve --start</code></p>
<p>However, when I submit requests to it like:</p>
<p><code>curl http://localhost:8080/predictions/my_tc/1.0 -T input_file.txt</code></p>
<p>It just hangs. No output.</p>
<ul>
<li>The <code>ping</code> API return Healthy for most of the time, and I assume after some time it hits a timeout and starts saying unhealthy.</li>
<li>I do see these errors popping up on the logs, I am not sure where they are coming from:</li>
</ul>
<pre><code class="lang-auto"> W-9009-my_tc_1.0-stdout MODEL_LOG - Backend worker process died.
my_tc_1.0-stdout MODEL_LOG - Traceback (most recent call last):
my_tc_1.0-stdout MODEL_LOG -   File ".../site-packages/ts/model_service_worker.py", line 250, in &lt;module&gt;
my_tc_1.0-stdout MODEL_LOG -     worker = TorchModelServiceWorker(
my_tc_1.0-stdout MODEL_LOG -   File ".../site-packages/ts/model_service_worker.py", line 69, in __init__
my_tc_1.0-stdout MODEL_LOG -     self.port = str(port_num) + LOCAL_RANK
my_tc_1.0-stdout MODEL_LOG - Traceback (most recent call last):
my_tc_1.0-stdout MODEL_LOG - TypeError: can only concatenate str (not "int") to str

</code></pre>
<ul>
<li>How can I provide the Context file via the command line?</li>
</ul>
<p>I suppose my questions are:</p>
<ul>
<li>When I do a <code>curl http://localhost:8080/predictions/my_tc/1.0 -T input_file.txt</code> What method is called? Which method in the Custom Handler or Base Handler is receiving the input file?</li>
<li>What should the input file look like? I think somewhere it was mentioned a Json?</li>
<li>Any examples which create the Context file and then pass it for serving?</li>
</ul>
<p>Thanks a lot!</p>

marksaroufim: <p>This is a known bug and it’s been fixed in the nightlies <a href="https://github.com/pytorch/serve/pull/2377" class="inline-onebox">fix tcp port in model_service_worker.py by lxning · Pull Request #2377 · pytorch/serve · GitHub</a></p>
<p>So in this case this issue is not your fault but if you’re trying to debug a model where the issue is probably yours the most useful to inspect <code>logs/model_log.log</code></p>

dhx: <p><a class="mention" href="/u/marksaroufim">@marksaroufim</a> - Thanks for the reply. Any ideas wrt these questions:</p>
<ul>
<li>
<p>When I do a <code>curl http://localhost:8080/predictions/my_tc/1.0 -T input_file.txt</code> What method is called?</p>
</li>
<li>
<p>Which method in the Custom Handler or Base Handler is receiving the input file?</p>
</li>
<li>
<p>What should the input file look like? I think somewhere it was mentioned a Json?<br>
Any examples which create the Context file and then pass it for serving?</p>
</li>
</ul>

lsmith77: <aside class="quote no-group" data-username="dhx" data-post="3" data-topic="182033">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/d/41988e/48.png" class="avatar"> dhx:</div>
<blockquote>
<p>Which method in the Custom Handler or Base Handler is receiving the input file?</p>
</blockquote>
</aside>
<p>From what I gather on <a href="https://pytorch.org/serve/custom_service.html" class="inline-onebox" rel="noopener nofollow ugc">Custom Service — PyTorch/Serve master documentation</a>, the first class is assumed to be the handler and then it calls the relevant functions listed on that page.</p>
<blockquote>
<p>NOTE - If you plan to have multiple classes in same python module/file then make sure that handler class is the first in the list</p>
</blockquote>

lsmith77: <aside class="quote no-group" data-username="marksaroufim" data-post="2" data-topic="182033">
<div class="title">
<div class="quote-controls"></div>
<img loading="lazy" alt="" width="24" height="24" src="https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/marksaroufim/48/36747_2.png" class="avatar"> marksaroufim:</div>
<blockquote>
<p>So in this case this issue is not your fault but if you’re trying to debug a model where the issue is probably yours the most useful to inspect <code>logs/model_log.log</code></p>
</blockquote>
</aside>
<p>If I don’t see anything showing up in that log, what could be the cause?</p>
<p>If I do a curl request while running “tail -f logs/*.log” I only see the following for my custom model. For the mnist example I see entries coming on for all the expected logs.</p>
<pre><code class="lang-auto">==&gt; logs/ts_log.log &lt;==
2023-06-20T07:38:36,108 [INFO ] epollEventLoopGroup-3-3 TS_METRICS - ts_inference_requests_total.Count:1.0|#model_name:EnglishContext,model_version:default|#hostname:app.0,timestamp:1687246716

==&gt; logs/ts_metrics.log &lt;==
2023-06-20T07:38:36,108 - ts_inference_requests_total.Count:1.0|#model_name:EnglishContext,model_version:default|#hostname:app.0,timestamp:1687246716
</code></pre>

marksaroufim: <p>It usually means there’s some critical failure with torchserve itself, which version are you using <a class="mention" href="/u/lsmith77">@lsmith77</a> ?</p>

lsmith77: <p>TorchServe Version is 0.8.1</p>
<p>Note the mnist model is working just fine.</p>

marksaroufim: <p>hmm <a class="mention" href="/u/lsmith77">@lsmith77</a> do you mind opening an issue on our github then? Our oncall can help you take a look</p>

lsmith77: <p>It suddenly started to work. I will try to figure out what I did to get it to work.</p>
<p>My custom handler is now called. Only issue is that I am returning a python list but I am always only getting the first element returned. I am using JSONEnvelope, so I am now looking at what is causing this issue. I assume it is related to the batching feature.</p>

lsmith77: <p>OK this to work … but now I am getting “Model "EnglishContext" has no worker to serve inference request. Please use scale workers API to add workers.” but I have no clue why this suddenly is happening.</p>
<p>I now added “default_workers_per_model=1” but it is still the case.</p>

lsmith77: <p>Alright things are working now. I think the root cause was model dependency related (and server running out of disk space). However I do not have a reproduce-able test case.</p>
<p><a class="mention" href="/u/marksaroufim">@marksaroufim</a> does it still make sense to open a ticket?</p>