Trouble deploying Yolov5 model in Torch Serve environment

username: atinesh
cooked: <p>I am trying to deploy the <a href="https://github.com/ultralytics/yolov5" rel="noopener nofollow ugc">Yolov5</a> model in the <a href="https://github.com/pytorch/serve" rel="noopener nofollow ugc">Torch Serve</a> environment but on sending the below request to the endpoint I am not getting any response, the request seems to get stuck</p>
<p><code>$ curl http://127.0.0.1:8080/predictions/yolov5 -T /home/atinesh/Desktop/COCO_val2014_000000562557.jpg</code></p>
<p><strong>Command to compile and deploy the model</strong></p>
<p>Starting container</p>
<pre><code class="lang-auto">$ sudo docker run -d --rm -it \
-p 8080:8080 -p 8081:8081 -p 8082:8082 \
--name torchserve-cpu \
-v $(pwd)/model-server/model-store:/home/model-server/model-store \
-v $(pwd)/model-server/examples:/home/model-server/examples \
pytorch/torchserve:latest
</code></pre>
<p>Logging in the container and compiling .mar file using Torch Model Archiver</p>
<pre><code class="lang-auto">$ sudo docker exec -u root -it torchserve-cpu /bin/bash

root@&lt;container-id&gt;:&lt;path&gt;# torch-model-archiver --model-name yolov5 \
--version 1.0 \
--serialized-file /home/model-server/examples/object_detector/yolov5/yolov5x.pt \
--handler /home/model-server/examples/object_detector/yolov5/torchserve_handler.py \
--export-path /home/model-server/model-store \
--extra-files /home/model-server/examples/object_detector/yolov5/torchserve_handler.py
</code></pre>
<p>Registering the model<br>
<code>$ curl -X POST "http://localhost:8081/models?url=/home/atinesh/model-server/model-store/yolov5.mar"</code></p>
<p>Scaling the worker<br>
<code>$ curl -v -X PUT "http://localhost:8081/models/yolov5?min_worker=1"</code></p>
<p>Deployed model is running properly, on checking model health I am getting below response</p>
<pre><code class="lang-auto">$ curl "http://localhost:8081/models/yolov5"
[
  {
    "modelName": "yolov5",
    "modelVersion": "1.0",
    "modelUrl": "/home/atinesh/model-server/model-store/yolov5.mar",
    "runtime": "python",
    "minWorkers": 1,
    "maxWorkers": 1,
    "batchSize": 1,
    "maxBatchDelay": 100,
    "loadedAtStartup": false,
    "workers": [
      {
        "id": "9001",
        "startTime": "2021-11-11T05:09:57.190Z",
        "status": "READY",
        "memoryUsage": 0,
        "pid": 507,
        "gpu": false,
        "gpuUsage": "N/A"
      }
    ]
  }
]
</code></pre>
<p><strong>Other information</strong></p>
<p><a href="https://github.com/ultralytics/yolov5" rel="noopener nofollow ugc">Yolov5</a> (XLarge) model is trained on custom <a href="https://cocodataset.org/#home" rel="noopener nofollow ugc">COCO</a> dataset to detect 2 objects person &amp; bicycle, below is the link of the trained model file<br>
<a href="https://www.dropbox.com/s/seu4ckvtuckhrpl/yolov5x.pt?dl=0" rel="noopener nofollow ugc">yolov5x.pt</a></p>
<p>Image used for Inference: <a href="https://www.dropbox.com/s/v4lst105hkfzv0n/coco_val2014_000000562557.jpg?dl=0" rel="noopener nofollow ugc">COCO_val2014_000000562557.jpg</a></p>
<p>I have tried two different handler files <a href="https://github.com/louisoutin/yolov5_torchserve/blob/master/ressources/torchserve_handler.py" rel="noopener nofollow ugc">handler #1</a>, <a href="https://gist.github.com/joek13/b895db0cd50a7c71a123611885057c69" rel="noopener nofollow ugc">handler #2</a> but the same issue persists.</p>

username: marksaroufim
cooked: <p>Hi <a class="mention" href="/u/atinesh">@atinesh</a> do you mind taking a look at the <code>logs/model_log.log</code> file and see if that helps debug the issue.</p>
<p>Some guesses as to the issue</p>
<p>My first thought was that inference time could be really slow but looks like model is ~150MB so should be fine, does this stuckness persist without torchserve? Can you try removing the batch delay and increasing the number of workers</p>
<p>Finally if you’re doing CPU inference it really helps to set <code>torch.set_num_threads(1)</code>  when initializing a model for infererence in your handler otherwise the inference can take a while. I’d also check whether the preprocessing is taking a long time so just some general profiling would help as well</p>